diff --git a/README.md b/README.md index d00da1f..9b25976 100644 --- a/README.md +++ b/README.md @@ -915,15 +915,15 @@ flowchart TD ### 4.1.3. Канонический MVP runtime (CODE-first) -Единая точка входа исполнения — пакет `app.modules.agent.runtime`: +Единая точка входа исполнения — пакет `app.core.agent.runtime`: -- **Роутер:** `app.modules.agent.intent_router_v2`; он отвечает и за routing, и за retrieval planning. -- **LLM-слой:** `app.modules.agent.llm`; здесь живут `AgentLlmService`, `PromptLoader` и системные prompt assets. -- **Runtime:** `app.modules.agent.runtime`; внутри него stages разложены по подпакетам `retrieval`, `context`, `gates`, `answer_policy`, `generation`, `finalization`. +- **Роутер:** `app.core.agent.intent_router`; он отвечает и за routing, и за retrieval planning. +- **LLM-слой:** `app.core.agent.llm`; здесь живут `AgentLlmService`, `PromptLoader` и системные prompt assets. +- **Runtime:** `app.core.agent.runtime`; внутри него stages разложены по подпакетам `retrieval`, `context`, `gates`, `answer_policy`, `generation`, `finalization`. - **Цепочка:** запрос → `IntentRouterV2` → retrieval planning → runtime retrieval adapter → нормализованный context/evidence → evidence gate 1 → answer policy → LLM generation → evidence gate 2 → finalization → diagnostics. - **Evidence gates:** pre/post проверки достаточности evidence и качества ответа по сценарию. - **Диагностика:** runtime возвращает machine-readable diagnostics и trace по стадиям. -- **RAG:** `app.modules.rag` больше не содержит agent use-case слоев; он остается инфраструктурой indexing/retrieval/storage. +- **RAG:** `app.core.rag` больше не содержит agent use-case слоев; он остается инфраструктурой indexing/retrieval/storage. Тесты: `pipeline_setup_v3` и связанные suite-ы проверяют канонический runtime и его stage-based execution. diff --git a/_process.zip b/_process.zip index 8577803..c954819 100644 Binary files a/_process.zip and b/_process.zip differ diff --git a/_process/03. Intents.md b/_process/03. Intents.md index e69de29..ebdc3dd 100644 --- a/_process/03. Intents.md +++ b/_process/03. Intents.md @@ -0,0 +1,59 @@ +# Intents + +## Domains + +- `DOCS` +- `GENERAL` +- `CODE` - временно отключен + +## GENERAL + +### Intent `GENERAL_QA` + +Общий интент для вопросов без точного маршрута. +В дальнейшем может использоваться как fallback. + +Subintents: +- `SUMMARY` - ответы на общие вопросы по SUMMARY + + + +## DOCS + +### Intent `ARCHITECTURE` + +Обработка вопросов по архитектуре. +Subintents пока отсутствуют. +Интент запланирован, без реализации. + +### Intent `DOC_EXPLAIN` + +Объяснение по документации. + +Subintents: +- `SUMMARY` - краткое объяснение темы по SUMMARY-блокам документации +- `FIND_FILES` - поиск файлов с релевантной информацией +- `EXPLAIN_API` - объяснение работы метода +- `COMPONENT_INTEGRATIONS` - перечень интеграций компонента, API, UI, сущности, внешних систем +- `ENTITY_INTEGRATIONS` - перечень интеграций сущности + +В текущем узком MVP реально реализованы только: + +- `SUMMARY` +- `FIND_FILES` + +Для запросов по интеграциям целевым retrieval-слоем является `D6_INTEGRATION_INDEX`. + +### Intent `OPENAPI_GENERATION` + +Генерация OpenAPI-спеки. + +Subintents: +- `FULL_SPEC` - создание полной спецификации + +### Intent `DOC_GENERATION` + +Редактирование документации. + +Subintents: +- `FROM_FEATURE` - создание документации из системной аналитики на фичу diff --git a/_process/03. RAG.md b/_process/03. RAG.md new file mode 100644 index 0000000..be00cc4 --- /dev/null +++ b/_process/03. RAG.md @@ -0,0 +1,356 @@ +# RAG + +## Состояние as is + +RAG сейчас используется как общее ядро индексации и retrieval по коду и документации. +Основной storage - `rag_session` и многослойный индекс в БД. + +## Основные части + +- `RagService` - фасад индексации и retrieval +- `DocsIndexingPipeline` - индексация документации +- `CodeIndexingPipeline` - индексация кода +- `RagRepository` - persistence и retrieval +- `IntentRouterV2` - планирование retrieval: слои, фильтры, ограничения +- `RuntimeRetrievalAdapter` - выполнение retrieval в runtime + +## Индексация + +Индексация идет по двум направлениям: + +- `DOCS` +- `CODE` + +На вход подается snapshot или changes. +Для каждого файла выбирается подходящий pipeline. +На выходе формируются документы по слоям и сохраняются в RAG-хранилище. + +## Структура БД + +Все слои сохраняются в общую таблицу `rag_chunks`. + +### Общие поля по слоям + +| Поле БД | Назначение | +|---|---| +| `rag_session_id` | идентификатор сессии индексации | +| `path` | путь исходного файла | +| `content` | основной текст записи для retrieval | +| `layer` | идентификатор слоя | +| `title` | короткий заголовок записи | +| `lang` | язык исходного содержимого, в основном для code-слоев | +| `repo_id` | идентификатор репозитория или проекта | +| `commit_sha` | версия кода или документов на момент индексации | +| `span_start`, `span_end` | диапазон строк в исходном файле, если он есть | +| `embedding` | векторное представление записи | +| `metadata_json` | структурированные атрибуты конкретного слоя | + +### Поля со смыслом слоя + +Смысл конкретного слоя хранится в `metadata_json`. +Именно эти атрибуты определяют, какой объект был извлечен и как его интерпретировать в retrieval. +Домены и поддомены должны храниться в `metadata_json` явно. + +## Слои DOCS + +### `D0_DOC_CHUNKS` + +Задача: +Хранит текстовые фрагменты документации для retrieval по содержимому разделов. + +Формирование: +Документ сначала разбирается на frontmatter и body, затем body режется на секции через markdown chunker. +Для каждой секции создается отдельная запись слоя. +Нарезка идет по разделам документа. +Только в fallback-сценарии, когда markdown-структура не найдена, используется нарезка по фиксированным текстовым чанкам. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `document_id` | идентификатор документа-источника | `frontmatter.id`, иначе путь файла | +| `type` | тип документа из frontmatter | `frontmatter.type` | +| `module` | модуль документа | `frontmatter.module` | +| `domain` | домен документа | `frontmatter.domain` | +| `subdomain` | поддомен документа | `frontmatter.subdomain` | +| `tags` | теги документа | `frontmatter.tags` | +| `section_path` | полный путь секции в иерархии заголовков | результат `MarkdownDocChunker` | +| `section_title` | заголовок текущей секции | результат `MarkdownDocChunker` | +| `order` | порядок секции внутри документа | результат `MarkdownDocChunker` | +| `doc_kind` | классификация документа, например `readme`, `spec`, `runbook` | `DocsClassifier.classify(path)` | +| `source_path` | исходный путь документа | путь файла | +| `artifact_type` | тип артефакта, здесь `DOCS` | константа builder | + +Связанные классы: +`DocsIndexingPipeline`, `DocsContentParser`, `MarkdownDocChunker`, `DocsDocumentBuilder` + +### `D1_DOCUMENT_CATALOG` + +Задача: +Хранит карточку документа как точку входа в документ и его краткое описание. + +Формирование: +Источник данных - frontmatter, fallback title, summary и doc kind, вычисленный классификатором документации. +Данные извлекаются структурированно по атрибутам. +В `content` попадает summary документа, а не склейка всех частей документа в сплошной текст. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `document_id` | идентификатор документа | `frontmatter.id`, иначе путь файла | +| `type` | тип документа из frontmatter | `frontmatter.type` | +| `name` | системное имя документа | `frontmatter.name` | +| `title` | человекочитаемый заголовок документа | `frontmatter.title`, иначе fallback title | +| `module` | модуль документа | `frontmatter.module` | +| `domain` | домен документа | `frontmatter.domain` | +| `subdomain` | поддомен документа | `frontmatter.subdomain` | +| `layer` | логический слой, указанный в frontmatter документа | `frontmatter.layer` | +| `status` | статус документа | `frontmatter.status` | +| `updated_at` | дата или отметка последнего обновления | `frontmatter.updated_at` | +| `tags` | теги документа | `frontmatter.tags` | +| `entities` | сущности, связанные с документом | `frontmatter.entities` | +| `parent` | родительский документ | `frontmatter.parent` | +| `children` | дочерние документы | `frontmatter.children` | +| `links` | ссылки на связанные материалы | `frontmatter.links` | +| `source_path` | исходный путь документа | путь файла | +| `summary_text` | краткое содержание документа | секция `# Summary` | +| `doc_kind` | классификация документа, например `readme`, `spec`, `runbook` | `DocsClassifier.classify(path)` | + +Связанные классы: +`DocsIndexingPipeline`, `DocsFrontmatterParser`, `DocsClassifier`, `DocsContentParser`, `DocsDocumentBuilder` + +### `D2_FACT_INDEX` + +Задача: +Хранит атомарные факты в форме `subject-predicate-object` для точного retrieval по утверждениям. + +Формирование: +Факты извлекаются из frontmatter и секций документа, после чего каждая найденная тройка превращается в отдельную запись. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `fact_id` | идентификатор факта | вычисляется builder из содержимого факта и пути | +| `subject_id` | субъект факта | `DocsFactExtractor` | +| `predicate` | предикат или тип связи | `DocsFactExtractor` | +| `object` | значение или объект факта | `DocsFactExtractor` | +| `object_ref` | ссылка на объект, если она выделена отдельно | `DocsFactExtractor` | +| `anchor` | место в документе, откуда взят факт | `DocsFactExtractor` | +| `tags` | теги факта | `DocsFactExtractor` | +| `source_path` | исходный путь документа | путь файла | + +Связанные классы: +`DocsIndexingPipeline`, `DocsFactExtractor`, `DocsDocumentBuilder` + +### `D3_ENTITY_CATALOG` + +Задача: +Хранит сущности, найденные в документации, чтобы искать документы и связи вокруг конкретной сущности. + +Формирование: +Сущности извлекаются из frontmatter документа, после чего каждая сущность сохраняется отдельной записью. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `entity_name` | имя сущности | `DocsEntityExtractor` | +| `document_id` | идентификатор документа, где найдена сущность | `frontmatter.id`, иначе путь файла | +| `document_type` | тип документа-источника | `frontmatter.type` | +| `module` | модуль документа | `frontmatter.module` | +| `domain` | домен документа | `frontmatter.domain` | +| `subdomain` | поддомен документа | `frontmatter.subdomain` | +| `tags` | теги документа или сущности | `frontmatter.tags` | +| `source_path` | исходный путь документа | путь файла | + +Связанные классы: +`DocsIndexingPipeline`, `DocsEntityExtractor`, `DocsDocumentBuilder` + +### `D4_WORKFLOW_INDEX` + +Задача: +Хранит workflow и сценарии из документации для ответов про flow, шаги и жизненный цикл процесса. + +Формирование: +Workflow извлекаются из detail sections документа и сохраняются как отдельные сценарии. +Извлечение идет из структуры `Details -> ## Сценарий`. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `workflow_id` | идентификатор сценария | вычисляется builder из названия, anchor и документа | +| `document_id` | идентификатор документа-источника | `frontmatter.id`, иначе путь файла | +| `workflow_name` | название сценария | блок `Details -> ## Сценарий -> **Название**` | +| `preconditions` | предусловия сценария | блок `Details -> ## Сценарий -> **Предусловия**` | +| `trigger` | триггер или событие запуска | блок `Details -> ## Сценарий -> **Триггер**` | +| `main_flow` | основной сценарий | блок `Details -> ## Сценарий -> **Основной сценарий**` | +| `alternative_flow` | альтернативные ветки | блок `Details -> ## Сценарий -> **Альтернативный сценарий**` | +| `error_handling` | обработка ошибок | блок `Details -> ## Сценарий -> **Обработка ошибок**` | +| `postconditions` | постусловия | блок `Details -> ## Сценарий -> **Постусловие**` | +| `source_path` | исходный путь документа | путь файла | + +Связанные классы: +`DocsIndexingPipeline`, `DocsWorkflowExtractor`, `DocsDocumentBuilder` + +### `D5_RELATION_GRAPH` + +Задача: +Хранит связи между документами и сущностями документации для navigation и related docs retrieval. + +Формирование: +Связи извлекаются из frontmatter и сохраняются как отдельные relation edges. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `relation_id` | идентификатор связи | вычисляется builder из source, target, relation type и anchor | +| `source_id` | источник связи | `frontmatter.id` или source документа в extractor | +| `relation_type` | тип связи | `DocsRelationExtractor` | +| `target_id` | целевой объект связи | `DocsRelationExtractor` | +| `anchor` | место в документе, где обнаружена связь | `DocsRelationExtractor` | +| `source_path` | исходный путь документа | путь файла | + +Связанные классы: +`DocsIndexingPipeline`, `DocsRelationExtractor`, `DocsDocumentBuilder` + +### `D6_INTEGRATION_INDEX` + +Задача: +Хранит прикладные интеграции компонента, API, UI, сущности или внешней системы. +Используется для ответов на вопросы вида "какие интеграции есть у компонента". + +Формирование: +Интеграции извлекаются из блока `## Integrations` документа. +Одна интеграция должна превращаться в отдельную запись слоя. +Описание интеграции может быть развернутым, а структурированные атрибуты должны выделяться в словарь. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `integration_id` | идентификатор интеграции | вычисляется builder из source, target и anchor | +| `source_id` | идентификатор объекта, для которого описана интеграция | `frontmatter.id` документа-источника | +| `source_type` | тип исходного объекта | `frontmatter.doc_type` | +| `target` | целевой объект интеграции | блок `## Integrations` | +| `target_type` | тип целевого объекта, например `api`, `ui`, `entity`, `service`, `external_system` | блок `## Integrations` | +| `direction` | направление интеграции | блок `## Integrations` | +| `interaction` | тип взаимодействия | блок `## Integrations` | +| `via` | технический канал интеграции | блок `## Integrations` | +| `purpose` | назначение интеграции | блок `## Integrations` | +| `details` | дополнительные атрибуты интеграции в виде словаря | блок `## Integrations` | +| `domain` | домен документа | `frontmatter.domain` | +| `subdomain` | поддомен документа | `frontmatter.subdomain` | +| `source_path` | исходный путь документа | путь файла | +| `anchor` | место в документе, где описана интеграция | блок `## Integrations` | + +Связанные классы: +`DocsIndexingPipeline`, `DocsIntegrationExtractor`, `DocsDocumentBuilder` + +## Слои CODE + +### `C0_SOURCE_CHUNKS` + +Задача: +Хранит фрагменты исходного кода как базовый слой для цитирования, explain и точечной догрузки кода. + +Формирование: +Исходный файл режется на кодовые чанки, и для каждого чанка создается отдельная запись. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `chunk_index` | порядковый номер чанка в файле | индекс чанка при нарезке | +| `chunk_type` | тип чанка, например функция, класс или текстовый блок | `CodeTextChunker` | +| `module_or_unit` | модуль, к которому относится chunk | вычисляется из пути файла | +| `is_test` | признак тестового файла | `is_test_path(path)` | + +Связанные классы: +`CodeIndexingPipeline`, `CodeTextChunker`, `CodeTextDocumentBuilder` + +### `C1_SYMBOL_CATALOG` + +Задача: +Хранит символы кода: классы, функции и методы. Используется для поиска по именам и структуре кода. + +Формирование: +Символы извлекаются `SymbolExtractor`, и каждый символ сохраняется как отдельная запись. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `symbol_id` | идентификатор символа | `SymbolExtractor` | +| `qname` | полное квалифицированное имя | `SymbolExtractor` | +| `kind` | тип символа: класс, функция, метод | `SymbolExtractor` | +| `signature` | сигнатура символа | `SymbolExtractor` | +| `parent_symbol_id` | родительский символ | `SymbolExtractor` | +| `package_or_module` | модуль или пакет символа | вычисляется из пути файла | +| `is_test` | признак тестового файла | `is_test_path(path)` | + +Связанные классы: +`CodeIndexingPipeline`, `PythonAstParser`, `SymbolExtractor`, `SymbolDocumentBuilder` + +### `C2_DEPENDENCY_GRAPH` + +Задача: +Хранит связи между символами кода: вызовы, импорты, наследование. Используется для анализа зависимостей и flow. + +Формирование: +Связи строятся `EdgeExtractor` по AST и списку символов, после чего каждая связь сохраняется отдельной записью. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `edge_id` | идентификатор связи | `EdgeExtractor` | +| `edge_type` | тип связи: вызов, импорт, наследование | `EdgeExtractor` | +| `src_symbol_id` | исходный символ | `EdgeExtractor` | +| `src_qname` | полное имя исходного символа | `EdgeExtractor` | +| `dst_symbol_id` | целевой символ, если он разрешен | `EdgeExtractor` | +| `dst_ref` | текстовая ссылка на целевой символ | `EdgeExtractor` | +| `resolution` | статус разрешения связи | `EdgeExtractor` | +| `is_test` | признак тестового файла | `is_test_path(path)` | + +Связанные классы: +`CodeIndexingPipeline`, `EdgeExtractor`, `EdgeDocumentBuilder` + +### `C3_ENTRYPOINTS` + +Задача: +Хранит точки входа приложения: HTTP routes, CLI commands и другие entrypoints. + +Формирование: +Детекторы ищут HTTP и CLI точки входа по символам файла, после чего каждый найденный entrypoint сохраняется отдельной записью. + +Фиксация в БД: +| Атрибут в `metadata_json` | Описание | Источник | +|---|---|---| +| `entry_id` | идентификатор точки входа | detector entrypoint model | +| `entry_type` | тип точки входа | detector entrypoint model | +| `framework` | framework, в котором найдена точка входа | detector entrypoint model | +| `route_or_command` | route или команда | detector entrypoint model | +| `handler_symbol_id` | идентификатор обработчика | detector entrypoint model | +| `handler_symbol` | имя обработчика | detector entrypoint model | +| `declaring_symbol` | символ, в котором объявлен entrypoint | detector entrypoint model | +| `entrypoint_kind` | вид точки входа | detector entrypoint model | +| `http_method` | HTTP-метод | detector entrypoint model | +| `route_path` | путь маршрута | detector entrypoint model | +| `decorator_text` | текст декоратора или объявления | detector entrypoint model | +| `summary_text` | краткое описание точки входа | detector entrypoint model | +| `is_test` | признак тестового файла | `is_test_path(path)` | +| `lang_payload` | дополнительные данные детектора | detector metadata | +| `artifact_type` | тип артефакта, здесь `CODE` | константа builder | + +Связанные классы: +`CodeIndexingPipeline`, `EntrypointDetectorRegistry`, `FastApiEntrypointDetector`, `FlaskEntrypointDetector`, `TyperClickEntrypointDetector`, `EntrypointDocumentBuilder` + +### `C4_SEMANTIC_ROLES` + +Задача: +Слой объявлен в enum и retrieval-планах как слой семантических ролей кода. + +Формирование: +Слой формируется на основе символов, связей, dataflow slices и execution traces. +В текущем runtime этот слой не используется как активный маршрут, так как домен `CODE` отключен. + +Фиксация в БД: +Смысловые атрибуты слоя сохраняются в `rag_chunks.metadata_json`. +Точное краткое описание состава этих атрибутов в текущем документе пока не зафиксировано. + +Связанные классы: +`CodeIndexingPipeline`, `SemanticRoleBuilder`, `SemanticRoleDocumentBuilder` diff --git a/_process/04. Analitycs artefacts.md b/_process/04. Analitycs artefacts.md index 74d9a83..2415f32 100644 --- a/_process/04. Analitycs artefacts.md +++ b/_process/04. Analitycs artefacts.md @@ -314,11 +314,71 @@ LLM не должна каждый раз тонуть в полном доку - функциональные требования; - UI; - API; +- integrations; - ошибки; - НФТ; - связи; - кодовые привязки. +### Блок `## Integrations` + +Если у объекта есть интеграции, они должны быть выделены в отдельный блок `## Integrations`. +Интеграции не нужно дублировать во frontmatter. +Основное описание хранится в body документа. + +Ожидаемый принцип: +- одна интеграция = одна отдельная запись внутри блока; +- у интеграции есть краткое имя; +- у интеграции есть структурированные атрибуты; +- дополнительные детали допускаются в свободной форме через вложенный словарь. + +Рекомендуемые атрибуты интеграции: +- `target` +- `target_type` +- `direction` +- `interaction` +- `via` +- `purpose` +- `details` + +Где: +- `target` - идентификатор или имя целевого объекта; +- `target_type` - тип цели: `api`, `ui`, `entity`, `service`, `queue`, `db`, `external_system`; +- `direction` - направление: `inbound`, `outbound`, `bidirectional`; +- `interaction` - тип взаимодействия: `calls`, `reads`, `writes`, `emits`, `consumes`, `depends_on`; +- `via` - технический канал интеграции; +- `purpose` - зачем нужна интеграция; +- `details` - словарь с гибкой структурой под дополнительные параметры. + +Пример: + +```md +## Integrations + +### Orders API +- target: api.orders.create +- target_type: api +- direction: outbound +- interaction: calls +- via: POST /api/orders +- purpose: создание заказа +- details: + - auth: service-token + - retry: true + +### Order Entity +- target: domain.order +- target_type: entity +- direction: outbound +- interaction: writes +- via: repository +- purpose: сохранение состояния заказа +- details: + - transaction: required +``` + +Этот блок должен быть пригоден и для чтения человеком, и для последующего извлечения в отдельный RAG-слой интеграций. + ## 1.10. Общие требования к markdown body 1. В документе должен быть один `H1`, совпадающий с `title`. @@ -428,6 +488,7 @@ UI-элементы должны храниться в **табличном** и ## Технический use case ## Функциональные требования ## Contract +## Integrations ## Errors ## Нефункциональные требования ## Связанные блоки логики @@ -454,6 +515,7 @@ UI-элементы должны храниться в **табличном** и ## Контекст ## Технический use case ## Функциональные требования +## Integrations ## Ограничения и условия вызова ## Нефункциональные требования ## Связанные API / UI / integration points @@ -787,4 +849,4 @@ DOCS и CROSS_DOMAIN остаются частью target architecture; в те - полноценные execution paths для всей системы; - богатые fact-индексы по всем доменам; - полный reference graph документации; -- глубокая автоматизация подготовки системной аналитики. \ No newline at end of file +- глубокая автоматизация подготовки системной аналитики. diff --git a/_process/05. MVP - process_v1.md b/_process/05. MVP - process_v1.md new file mode 100644 index 0000000..51c3d8a --- /dev/null +++ b/_process/05. MVP - process_v1.md @@ -0,0 +1,100 @@ +# MVP: процесс v1 + +## 1. Общее описание + +Запрос пользователя обрабатывается цепочкой API → рантайм агента → зарегистрированный процесс версии `v1` → один workflow из трёх последовательных шагов. Процесс **не** обращается к RAG и **не** маршрутизирует интенты: текст сообщения передаётся в LLM по фиксированному промпту. Ответ агента — результат генерации с лёгкой постобработкой (trim). + +```mermaid +flowchart LR + subgraph api [API] + RS[RequestService] + end + subgraph runtime [Agent runtime] + AR[AgentRuntime] + PR[ProcessRunner] + end + subgraph v1 [Процесс v1] + P1[V1Process] + WG[V1FlowMainGraph] + end + subgraph wf [Workflow v1.flow_main] + S1[PrepareUserMessageStep] + S2[GenerateAnswerStep] + S3[FinalizeAnswerStep] + end + LLM[AgentLlmService] + RS --> AR + AR --> PR + PR --> P1 + P1 --> WG + WG --> S1 --> S2 --> S3 + S2 --> LLM +``` + +Клиент создаёт запрос с `process_version: v1`. `AgentRuntime` поднимает `RuntimeExecutionContext` (запрос, сессия, publisher, trace), выбирает `V1Process` из реестра и вызывает `run`. `V1Process` собирает `V1FlowContext` и прогоняет линейный граф: подготовка текста, один вызов LLM, финализация строки ответа. Итог попадает в `ProcessResult.answer` и дальше в ответ пользователю. + +--- + +## 2. Шаги и контракты + +### 2.1. Вход в процесс: `V1Process.run` + +| | | +|--|--| +| **Название** | Запуск процесса v1 | +| **Задача** | Собрать контекст workflow и выполнить граф до готового ответа. | +| **Вход** | `RuntimeExecutionContext`: `request` (в т.ч. `message`), `session`, `publisher`, `trace`. | +| **Выход** | `ProcessResult` с полем `answer: str`. | +| **Как работает** | Создаётся `V1FlowContext` с `prompt_name` по умолчанию `v1_flow_main.answer`. Вызывается `V1FlowMainGraph.run`. Возвращается ответ из контекста workflow. | + +Код: `src/app/core/agent/processes/v1/process.py`. + +--- + +### 2.2. Шаг workflow: `PrepareUserMessageStep` + +| | | +|--|--| +| **Название** | Подготовка сообщения пользователя | +| **Задача** | Сформировать строку, которая уйдёт в LLM как пользовательский ввод. | +| **Вход** | `V1FlowContext` с заполненным `runtime` и `prompt_name`. | +| **Выход** | Тот же контекст с `prepared_message: str`. | +| **Как работает** | Берётся `context.runtime.request.message` и обрезаются пробелы по краям (`strip`). Результат пишется в `prepared_message`. Других преобразований нет. | + +Код: `src/app/core/agent/processes/v1/workflow/flow_main/steps/prepare_user_message_step.py`. + +--- + +### 2.3. Шаг workflow: `GenerateAnswerStep` + +| | | +|--|--| +| **Название** | Вызов LLM | +| **Задача** | Сгенерировать ответ по выбранному промпту и подготовленному сообщению. | +| **Вход** | `V1FlowContext` с `prepared_message`, `prompt_name`, `runtime.trace` для модуля LLM. | +| **Выход** | Контекст с `answer: str` (сырой ответ модели). | +| **Как работает** | Асинхронно в пуле потоков вызывается `AgentLlmService.generate(prompt_name, prepared_message, ...)`. В trace подключается модуль `workflow.v1.llm`. Идентификатор запроса передаётся в `log_context` для логов. | + +Код: `src/app/core/agent/processes/v1/workflow/flow_main/steps/generate_answer_step.py`. + +--- + +### 2.4. Шаг workflow: `FinalizeAnswerStep` + +| | | +|--|--| +| **Название** | Финализация ответа | +| **Задача** | Нормализовать строку ответа перед выдачей пользователю. | +| **Вход** | `V1FlowContext` с заполненным `answer` после LLM. | +| **Выход** | Контекст с обновлённым `answer`. | +| **Как работает** | К ответу применяется `strip()` по краям. Другой логики нет. | + +Код: `src/app/core/agent/processes/v1/workflow/flow_main/steps/finalize_answer_step.py`. + +--- + +### 2.5. Транспорт: `WorkflowGraph` (v1) + +Граф для v1 использует стандартный `WorkflowGraph`: на каждом шаге пишутся события `workflow_started`, `step_started`, `step_completed`, `workflow_completed` в `runtime_traces` через `context.runtime.trace`. + +Код: `src/app/core/agent/utils/workflow/graph.py`, обёртка `V1FlowMainGraph` в `src/app/core/agent/processes/v1/workflow/flow_main/graph.py`. diff --git a/_process/05. MVP - process_v2.md b/_process/05. MVP - process_v2.md new file mode 100644 index 0000000..b019846 --- /dev/null +++ b/_process/05. MVP - process_v2.md @@ -0,0 +1,220 @@ +# MVP: процесс v2 + +## 1. Общее описание + +Процесс v2 в текущем MVP ориентирован в первую очередь на **документацию проекта**, но роутер также поддерживает `GENERAL / GENERAL_QA / SUMMARY` для общих обзорных вопросов. Для документных веток нужна активная RAG-сессия с проиндексированными документами. + +Это **узкий MVP**, а не полная target architecture. Поддерживаются три маршрута: + +- `GENERAL` +- `GENERAL_QA` +- `SUMMARY` +- `DOCS` +- `DOC_EXPLAIN` +- `SUMMARY` +- `FIND_FILES` + +Запрос проходит следующие смысловые этапы: + +1. проверка готовности сессии; +2. intent routing; +3. формирование retrieval-параметров; +4. retrieval из `DOCS RAG`; +5. минимальная сборка evidence; +6. запуск task-focused workflow нужной ветки; +7. формирование ответа. + +```mermaid +flowchart TB + subgraph api [API] + RS[RequestService] + end + subgraph runtime [Agent runtime] + AR[AgentRuntime] + PR[ProcessRunner] + end + subgraph v2 [Процесс v2] + P2[V2Process] + IR[V2IntentRouter] + POL[V2RetrievalPolicyResolver] + AD[V2RagRetrievalAdapter] + RSR[RagSessionRetriever] + ASM[DocsEvidenceAssembler] + end + subgraph rag [Пакет rag] + RR[RagRepository] + end + subgraph wf [Workflow] + SUM[DocsExplainSummaryGraph] + FF[DocsExplainFindFilesGraph] + end + LLM[AgentLlmService] + RS --> AR --> PR --> P2 + P2 --> IR --> POL --> AD --> RSR --> RR + AD --> ASM + ASM --> SUM + ASM --> FF + SUM --> LLM +``` + +Клиент указывает `process_version: v2`. Без `active_rag_session_id` в сессии процесс возвращает сообщение об ошибке. Иначе выполняется цепочка: + +маршрутизация → `RetrievalPlan` → retrieval строк из `DOCS RAG` → минимальная сборка evidence → ветвление по `subintent` → запуск workflow. + +### Реализованные домены, интенты и сабинтенты + +В коде заданы константы `V2Domain`, `V2Intent`, `V2Subintent`. Сейчас процесс intentionally ограничен одной рабочей областью. + +| Уровень | Значение (строка) | Реализация | +|--------|-------------------|------------| +| **Домен (routing_domain)** | `DOCS` | Единственный поддерживаемый домен: документация проекта. | +| **Интент** | `DOC_EXPLAIN` | Единственный интент: объяснение по документации. | +| **Сабинтент** | `SUMMARY` | Объяснение темы по SUMMARY-блокам документации. | +| **Сабинтент** | `FIND_FILES` | Поиск путей к документам, где описана нужная сущность или тема. | + +Итого в текущем MVP реализована **одна** рабочая тройка домен×интент: `DOCS` + `DOC_EXPLAIN`, с **двумя** ветками по сабинтенту. + +--- + +## 2. Этапы вне workflow (внутри `V2Process.run`) + +### 2.1. `V2IntentRouter.route` + +| | | +|--|--| +| **Название** | Маршрутизация запроса (v2) | +| **Задача** | Определить домен, интент, subintent и извлечь якоря из текста. | +| **Вход** | `user_query: str` (текст сообщения пользователя). | +| **Выход** | `V2RouteResult`: `routing_domain`, `intent`, `subintent`, `user_query`, `normalized_query`, `target_terms`, `anchors` (`V2RouteAnchors`), `confidence`. | +| **Как работает** | Router реализован по схеме **LLM-first**: `normalization` → `target_terms`/`anchors extraction` → `LLM router` → `deterministic validator` → `fallback`. LLM является **основным селектором маршрута**. Deterministic-слой больше не выбирает маршрут по умолчанию: он отвечает только за extraction, валидацию enum/комбинаций и fallback при сломанном или невалидном ответе LLM. В trace пишется событие `intent_routed`. | + +Код: `src/app/core/agent/processes/v2/intent_router/router.py`, `modules/normalizer.py`, `modules/target_terms.py`, `modules/anchors.py`, `routers/llm.py`, `routers/validator.py`, `routers/fallback.py`. + +--- + +### 2.2. `V2RetrievalPolicyResolver.resolve` + +| | | +|--|--| +| **Название** | Политика retrieval для v2 | +| **Задача** | По результату роутинга выбрать профиль, список слоёв RAG и лимит строк выдачи. | +| **Вход** | `V2RouteResult`. | +| **Выход** | `RetrievalPlan`: `profile`, `layers`, `limit`, опционально `filters`. | +| **Как работает** | Это отдельный смысловой шаг между routing и retrieval. Он не ходит в БД и не извлекает данные, а только подготавливает параметры поиска. Для `FIND_FILES` выбирается один профиль слоёв и лимит, для `SUMMARY` — другой. Лог: `retrieval_plan_resolved`. | + +Код: `src/app/core/agent/processes/v2/retrieval/policy_resolver.py`. + +--- + +### 2.3. `V2RagRetrievalAdapter` → `RagSessionRetriever.retrieve` + +| | | +|--|--| +| **Название** | Загрузка сырых строк из RAG по плану | +| **Задача** | Делегировать поиск в единственную реализацию retrieval в пакете `rag`. | +| **Вход** | `rag_session_id`, `query_text` (нормализованный запрос), `RetrievalPlan`. | +| **Выход** | `list[dict]` — строки чанков в формате `RagRepository.retrieve` (поля `path`, `layer`, `metadata`, и т.д.). | +| **Как работает** | Выполняется retrieval по уже сформированному плану: профиль, список слоёв и лимит. На этом шаге происходит только извлечение сырых строк из `DOCS RAG`. Лог: `rag_rows_fetched`. | + +Код адаптера: `src/app/core/agent/processes/v2/retrieval/v2_rag_adapter.py`. +Код API: `src/app/core/rag/retrieval/session_retriever.py`. + +--- + +### 2.4. `DocsEvidenceAssembler` + +| | | +|--|--| +| **Название** | Сборка evidence для задачи | +| **Задача** | Превратить сырые строки retrieval в списки summary или кандидатов файлов с дедупом и скорингом. | +| **Вход** | Список строк `rows`, `V2RouteResult` (для `target_terms`). | +| **Выход** | `list[RetrievedSummary]` или `list[RetrievedFile]`. | +| **Как работает** | Это **минимальная evidence-проверка**, достаточная для MVP. Для `SUMMARY` отбрасываются записи без summary-текста и summary-like секции, затем применяется дедуп и простой скоринг по терминам. Для `FIND_FILES` остаются только релевантные пути документов, также с дедупом и простым скорингом. Здесь нет сложной многоступенчатой валидации: задача шага — отфильтровать очевидный шум и передать в workflow компактное evidence. Лог: `evidence_assembled`. | + +Код: `src/app/core/agent/processes/v2/evidence/assembler.py`. + +--- + +## 3. Шаги workflow + +Текущие workflow являются **task-focused**: каждая ветка решает одну узкую прикладную задачу и не содержит общей универсальной логики для всех типов вопросов. + +### 3.1. Ветка `SUMMARY`: `GenerateSummaryAnswerStep` + +| | | +|--|--| +| **Название** | Сборка ответа по summary | +| **Задача** | Сформировать ответ пользователю по найденным SUMMARY-блокам или сообщить об отсутствии. | +| **Вход** | `DocsExplainSummaryContext`: `runtime`, `route`, `rag_session_id`, `prompt_name`, `documents` (список `RetrievedSummary`). | +| **Выход** | Контекст с `answer: str`, `prompt_input` при успешном вызове LLM. | +| **Как работает** | Workflow получает уже отобранные summary-документы. Если документов нет — возвращает честный fallback-ответ. Иначе собирает prompt input из запроса пользователя и найденных summary-блоков и вызывает LLM. Workflow не занимается retrieval и не строит retrieval-план: он решает только задачу генерации ответа по уже подготовленному evidence. | + +Код: `src/app/core/agent/processes/v2/workflows/docs_explain_summary/steps/generate_summary_answer_step.py`. +Граф: `DocsExplainSummaryGraph` (`V2WorkflowGraph`). + +--- + +### 3.2. Ветка `FIND_FILES`: `FinalizeFindFilesAnswerStep` + +| | | +|--|--| +| **Название** | Сборка списка файлов | +| **Задача** | Вывести пользователю markdown-список путей к файлам документации. | +| **Вход** | `DocsExplainFindFilesContext`: `runtime`, `route`, `rag_session_id`, `files` (`RetrievedFile`). | +| **Выход** | Контекст с `answer: str`. | +| **Как работает** | Workflow получает уже собранный список файлов и формирует финальный ответ. Если файлов нет — возвращает fallback. Если файлы есть — отдает детерминированный список путей. Эта ветка intentionally не использует LLM, потому что задача сводится к выдаче путей, а не к генерации объяснения. | + +Код: `src/app/core/agent/processes/v2/workflows/docs_explain_find_files/steps/finalize_find_files_answer_step.py`. +Граф: `DocsExplainFindFilesGraph` (`V2WorkflowGraph`). + +--- + +### 3.3. Транспорт: `V2WorkflowGraph` + +| | | +|--|--| +| **Название** | Workflow v2 с буфером trace | +| **Задача** | Выполнить шаги без пошаговых `step_started`/`step_completed` в trace; один раз сбросить сводку. | +| **Вход** | Контекст workflow (`DocsExplainSummaryContext` или `DocsExplainFindFilesContext`). | +| **Выход** | Обновлённый контекст. | +| **Как работает** | Для каждого шага: `trace_input` до `run`, затем `run`, затем `trace_output`; записи копятся в список. В trace уходят `workflow_started`, затем `workflow_trace_flushed` с массивом шагов, затем `workflow_completed`. Статусы пользователю публикуются через `publisher` как и раньше. | + +Код: `src/app/core/agent/processes/v2/workflows/v2_workflow_graph.py`. + +--- + +## 4. Сборка в приложении + +В `ModularApplication` создаются `RagSessionRetriever`, `V2RagRetrievalAdapter`, `V2RetrievalPolicyResolver`, `DocsEvidenceAssembler` и передаются в `V2Process` (см. `src/app/core/application.py`). + +--- + +## 5. Итоговая концептуальная схема текущего MVP + +В концептуальном виде текущий `v2` работает так: + +1. **Session check** + Проверка, что есть активная RAG-сессия проекта. + +2. **LLM-first intent routing** + Нормализация, extraction (`target_terms`, `anchors`), затем основной выбор маршрута через LLM. + +3. **Deterministic validation + fallback** + Проверка enum/комбинации маршрута и fallback только если LLM не ответил или вернул невалидный маршрут. + +4. **Retrieval parameter planning** + Формирование профиля поиска, слоёв и лимитов. + +5. **RAG retrieval** + Загрузка сырых строк из `DOCS RAG`. + +6. **Minimal evidence assembly** + Дедуп, базовый скоринг, отбор полезных summary или файлов. + +7. **Task-focused workflow** + Узкая ветка `SUMMARY` или `FIND_FILES`. + +8. **Final response** + Либо explanation через LLM, либо детерминированный список файлов. + +Это и есть актуальная архитектура **узкого MVP**, синхронизированная с текущей реализацией. diff --git a/_process/components/v2_intent_router_architecture.md b/_process/components/v2_intent_router_architecture.md new file mode 100644 index 0000000..5ecb641 --- /dev/null +++ b/_process/components/v2_intent_router_architecture.md @@ -0,0 +1,269 @@ +# V2IntentRouter Architecture + +## 1. Архитектура + +Текущий `V2IntentRouter` состоит из следующих компонентов: + +- `router.py` + Главная точка входа и оркестратор. + +- `modules/normalizer.py` + Нормализация текста запроса в `normalized_query`. + +- `modules/target_terms.py` + Извлечение `target_terms`, `endpoint_paths`, `matched_aliases`, `alias_docs`. + +- `modules/anchors.py` + Извлечение `anchors` и вспомогательных marker-сигналов. + +- `routers/docs_subintent_resolver.py` + Определение `subintent`. + +- `routers/deterministic.py` + Детерминированное определение `routing_domain`, `intent`, `subintent`, `confidence`, `routing_mode`, `llm_router_used`, `reason_short`. + +- `routers/llm.py` + LLM-based определение `routing_domain`, `intent`, `subintent`, `confidence`, `reason_short`. + +- `routers/prompts.yml` + Prompt для LLM-router. + +## 2. Контракт + +### Вход + +- `user_query: str` + +### Выход + +`V2RouteResult`: + +- `routing_domain: str` +- `intent: str` +- `subintent: str` +- `user_query: str` +- `normalized_query: str` +- `target_terms: list[str]` +- `anchors: V2RouteAnchors` +- `confidence: float` +- `routing_mode: str` +- `llm_router_used: bool` +- `reason_short: str` + +`V2RouteAnchors`: + +- `entity_names: list[str]` +- `terms: list[str]` +- `file_names: list[str]` +- `endpoint_paths: list[str]` +- `target_doc_hints: list[str]` +- `matched_aliases: list[str]` +- `process_domain: str | None` +- `process_subdomain: str | None` + +## 3. Поддерживаемые домены, интенты и сабинтенты + +### Домены + +- `DOCS` +- `GENERAL` + +### Интенты + +- `DOC_EXPLAIN` +- `GENERAL_QA` + +### Сабинтенты + +- `SUMMARY` +- `FIND_FILES` + +### Поддерживаемые маршруты + +- `GENERAL / GENERAL_QA / SUMMARY` +- `DOCS / DOC_EXPLAIN / SUMMARY` +- `DOCS / DOC_EXPLAIN / FIND_FILES` + +## 4. Флоу обработки запроса + +1. `router.py` принимает `user_query`. +2. `modules/normalizer.py` строит `normalized_query`. +3. `modules/target_terms.py` извлекает ключевые термы и alias-based сигналы. +4. `modules/anchors.py` строит `anchors` и marker-сигналы. +5. `router.py` собирает `QueryFeatures`. +6. `routers/deterministic.py` пытается определить маршрут детерминированно. +7. Если deterministic route найден, он сразу возвращается. +8. Если deterministic route не найден, `router.py` вызывает `routers/llm.py`. +9. Если LLM вернул валидный маршрут, собирается `V2RouteResult` с `routing_mode="llm_assisted"`. +10. Если LLM недоступен или не вернул валидный маршрут, используется fallback: + `GENERAL / GENERAL_QA / SUMMARY` с `routing_mode="llm_fallback"`. + +## 5. Компоненты по флоу + +### `router.py` + +- Задача + Собрать весь процесс роутинга в одной входной точке. + +- Как решает + Последовательно вызывает normalizer, target terms extractor, anchors extractor, deterministic router и при необходимости LLM router. + +- Вход + `user_query: str` + +- Выход + `V2RouteResult` + +### `modules/normalizer.py` + +- Задача + Привести запрос к стабильной форме для дальнейшего анализа. + +- Как решает + Схлопывает лишние пробелы через `" ".join(...split())`. + +- Вход + `user_query: str` + +- Выход + `normalized_query: str` + +### `modules/target_terms.py` + +- Задача + Выделить ключевые термы и retrieval-сигналы из запроса. + +- Как решает + Использует: + - regex для path/entity-like фрагментов + - список stop-words + - alias rules с фразами и каноническими термами + - эвристику для `/health` + +- Вход + `normalized_query: str` + +- Выход + `TargetTermsAnalysis`: + - `target_terms` + - `endpoint_paths` + - `matched_aliases` + - `alias_docs` + +### `modules/anchors.py` + +- Задача + Построить полный набор `anchors` и doc-oriented marker-сигналов. + +- Как решает + Использует: + - regex для `entity_names` и `file_names` + - словари marker-фраз: + - file markers + - architecture markers + - logic markers + - domain markers + - endpoint markers + - map `endpoint -> target_doc_hint` + - alias docs из `TargetTermsAnalysis` + +- Вход + - `normalized_query: str` + - `TargetTermsAnalysis` + +- Выход + `AnchorAnalysis`: + - `anchors` + - `file_markers` + - `architecture_markers` + - `logic_markers` + - `domain_markers` + - `endpoint_markers` + +### `routers/docs_subintent_resolver.py` + +- Задача + Определить `subintent`. + +- Как решает + Эвристика: + - если есть `file_markers` -> `FIND_FILES` + - если есть doc-signals (`endpoint_paths`, `endpoint_markers`, `architecture_markers`, `logic_markers`, `domain_markers`, `target_doc_hints`) -> `SUMMARY` + - иначе `None` + +- Вход + `QueryFeatures` + +- Выход + `subintent: str | None` + +### `routers/deterministic.py` + +- Задача + Детерминированно определить маршрут без LLM там, где это возможно. + +- Как решает + Использует: + - `DocsSubintentResolver` + - проверку conflicting doc anchors + - список general markers + + Правила: + - `FIND_FILES` -> `DOCS / DOC_EXPLAIN / FIND_FILES` + - `subintent != None` и нет конфликта doc-signals -> `DOCS / DOC_EXPLAIN / SUMMARY` + - general marker -> `GENERAL / GENERAL_QA / SUMMARY` + +- Вход + - `user_query: str` + - `QueryFeatures` + - `anchors: V2RouteAnchors` + +- Выход + `V2RouteResult | None` + +### `routers/llm.py` + +- Задача + Определить маршрут через LLM, если deterministic routing не дал результата. + +- Как решает + Формирует JSON payload из: + - `user_query` + - `normalized_query` + - `target_terms` + - `anchors` + - списка допустимых маршрутов + + Затем: + - вызывает LLM + - парсит JSON + - валидирует маршрут по whitelist + - нормализует `confidence` + +- Вход + - `user_query: str` + - `normalized_query: str` + - `target_terms: list[str]` + - `anchors: dict` + +- Выход + `dict | None`: + - `routing_domain` + - `intent` + - `subintent` + - `confidence` + - `reason_short` + +### `routers/prompts.yml` + +- Задача + Задать LLM-router формальный контракт ответа. + +- Как решает + Описывает допустимые маршруты и требует вернуть только JSON. + +- Вход + Payload от `routers/llm.py` + +- Выход + Структурированный JSON-ответ LLM diff --git a/docs/documentation/rag/architecture-overview.md b/docs/documentation/rag/architecture-overview.md index 0493417..e0b5b41 100644 --- a/docs/documentation/rag/architecture-overview.md +++ b/docs/documentation/rag/architecture-overview.md @@ -37,11 +37,11 @@ tags: - Scope: модуль индексации проектных файлов, хранения RAG-слоёв и выдачи retrieval-контекста. - Purpose: построить индекс по документации и Python-коду и дать runtime доступ к релевантным фрагментам. -- Main modules: `RagModule`, `RagService`, `IndexingOrchestrator`, `RagRepository`, `RepoWebhookService`. +- Main modules: `RagModule`, `RagService`, `IndexingOrchestrator`, `RagRepository`. - Main domains: RAG-сессии, задачи индексации, документы индекса, blob-cache, retrieval. - Main integrations: PostgreSQL/pgvector, GigaChat embeddings, FastAPI, EventBus, story context. -- Key entrypoints: `/api/rag/sessions`, `/api/rag/sessions/{rag_session_id}/changes`, `/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}`, `/internal/rag-repo/webhook`. -- Key data flows: snapshot indexing, incremental reindex, retrieval из `rag_chunks`, webhook-нормализация коммитов. +- Key entrypoints: `/api/rag/sessions`, `/api/rag/sessions/{rag_session_id}/changes`, `/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}`, `/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}/events`. +- Key data flows: snapshot indexing, incremental reindex, retrieval из `rag_chunks`. - Source of truth: код `src/app/modules/rag/*`. ## Назначение @@ -50,7 +50,7 @@ tags: ## Контекст -Модуль используется как инфраструктурный слой для agent/runtime и смежных интеграций. На вход он принимает либо список файлов проекта, либо webhook репозитория. На выходе формирует устойчивый индекс, ассоциированный с `rag_session_id`, и статус задач индексации, пригодный для опроса и SSE-подписки. +Модуль используется как инфраструктурный слой для agent/runtime. На вход он принимает snapshot и изменения файлов проекта. На выходе формирует устойчивый индекс, ассоциированный с `rag_session_id`, и статус задач индексации, пригодный для опроса и SSE-подписки. ## Границы системы @@ -74,7 +74,7 @@ tags: ## Архитектурная схема -`RagModule` собирает зависимости модуля и публикует HTTP endpoints. Для индексации он использует `RagSessionStore`, `IndexJobStore`, `IndexingOrchestrator` и `RagService`. `RagService` выбирает docs/code pipeline, обогащает документы метаданными файла, запрашивает embeddings и записывает результат через `RagRepository`. `RagRepository` агрегирует schema/session/job/document/cache/query репозитории. Отдельно `RagRepoModule` принимает repository webhooks и прокидывает нормализованный commit context в story storage и cache writer. +`RagModule` собирает зависимости модуля и публикует HTTP endpoints. Для индексации он использует `RagSessionStore`, `IndexJobStore`, `IndexingOrchestrator` и `RagService`. `RagService` выбирает docs/code pipeline, обогащает документы метаданными файла, запрашивает embeddings и записывает результат через `RagRepository`. `RagRepository` агрегирует schema/session/job/document/cache/query репозитории. ## Основные модули @@ -87,7 +87,6 @@ tags: | `DocsIndexingPipeline` | построение слоёв документации `D1-D4` | classifier, chunker, document builder | `src/app/modules/rag/indexing/docs/pipeline.py` | | `CodeIndexingPipeline` | построение слоёв кода `C0-C4` | AST parser, symbol/edge/entrypoint/role builders | `src/app/modules/rag/indexing/code/pipeline.py` | | `RagRepository` | единая точка persistence и retrieval | schema/session/job/document/cache/query repositories | `src/app/modules/rag/persistence/repository.py` | -| `RepoWebhookService` | нормализация webhook payload и выделение story id | story writer, cache writer | `src/app/modules/rag/webhook_service.py` | ## Основные доменные области @@ -104,9 +103,8 @@ tags: | ------------------------ | --------- | --------------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------- | | PostgreSQL + pgvector | outbound | хранение документов, jobs, sessions и vector search | SQLAlchemy / SQL / pgvector | `logic-rag-retrieval` | | GigaChat embeddings | outbound | получение embedding для batch документов | HTTP client через `GigaChatClient` | `logic-rag-indexing` | -| FastAPI | inbound | публичный и internal API модуля | HTTP | `api-rag-session-create`, `api-rag-session-changes`, `api-rag-session-job` | +| FastAPI | inbound | публичный HTTP API модуля | HTTP | `api-rag-session-create`, `api-rag-session-changes`, `api-rag-session-job` | | EventBus | outbound | публикация прогресса индексации и terminal events | in-process async events / SSE | `api-rag-session-job` | -| Story context repository | outbound | запись webhook-коммитов для story | Python interface | `logic-rag-indexing` | ## Основные потоки @@ -138,7 +136,7 @@ tags: - Code indexing поддерживает только Python-файлы. - Docs indexing ориентирован на markdown и frontmatter YAML. -- Deprecated endpoint `/internal/rag/retrieve` не используется для рабочего retrieval. +- HTTP retrieval endpoint в модуле не публикуется. - Реальное retrieval API доступно через repository/runtime adapters, а не через публичный HTTP endpoint модуля. ### Risks @@ -152,7 +150,6 @@ tags: ### Security - Публичные endpoints не содержат собственной бизнес-авторизации внутри модуля и полагаются на внешний слой приложения. -- Webhook provider определяется по headers/payload без явной проверки подписи в самом `RepoWebhookService`. ### Reliability @@ -164,7 +161,7 @@ tags: - Logs: `RagService` пишет предупреждения по cache hit/miss и skipped files. - Metrics: явные метрики не выделены. - Traces: явная трассировка не реализована. -- Audit: job status и webhook commit binding сохраняются в БД. +- Audit: job status сохраняется в БД. ### Performance @@ -191,16 +188,13 @@ tags: - `src/app/modules/rag/indexing_service.py` - `src/app/modules/rag/persistence/repository.py` - `src/app/modules/rag/persistence/schema_repository.py` -- `src/app/modules/rag/webhook_service.py` ### Symbols - `RagModule` -- `RagRepoModule` - `RagService` - `IndexingOrchestrator` - `RagRepository` -- `RepoWebhookService` ## Связанные документы @@ -218,5 +212,3 @@ tags: | Date | Source | Changes | | ---------- | ------ | ------------------------------------------------------------------- | | 2026-03-13 | code | Создан обзор архитектуры пакета `rag` на основе текущей реализации. | - - diff --git a/docs/documentation/rag/logic-indexing-pipeline.md b/docs/documentation/rag/logic-indexing-pipeline.md index 8559f61..1f71c2a 100644 --- a/docs/documentation/rag/logic-indexing-pipeline.md +++ b/docs/documentation/rag/logic-indexing-pipeline.md @@ -125,8 +125,6 @@ tags: - `POST /api/rag/sessions` - `POST /api/rag/sessions/{rag_session_id}/changes` -- `POST /internal/rag/index/snapshot` -- `POST /internal/rag/index/changes` ## Связанные сущности diff --git a/docs/documentation/rag/logic-retrieval-ranking.md b/docs/documentation/rag/logic-retrieval-ranking.md index 720b6dd..91ca390 100644 --- a/docs/documentation/rag/logic-retrieval-ranking.md +++ b/docs/documentation/rag/logic-retrieval-ranking.md @@ -90,7 +90,7 @@ tags: - Retrieval работает только внутри одной `rag_session_id` и не агрегирует несколько сессий. - Layer ranking зашит в код SQL-builder и требует явного обновления при появлении новых слоёв. -- Полноценный HTTP retrieval endpoint в модуле отсутствует: `/internal/rag/retrieve` возвращает `410 deprecated`. +- Полноценный HTTP retrieval endpoint в модуле не публикуется. ## Нефункциональные требования @@ -115,7 +115,7 @@ tags: - Runtime retrieval adapters в `src/app/modules/agent/runtime/steps/retrieval/adapter.py` - Explain retrieval gateway в `src/app/modules/agent/runtime/steps/explain/layered_gateway.py` -- Deprecated endpoint `POST /internal/rag/retrieve` +- HTTP retrieval endpoint отсутствует ## Связанные сущности diff --git a/docs/intent-router-mvp.md b/docs/intent-router-mvp.md new file mode 100644 index 0000000..a8ca5ae --- /dev/null +++ b/docs/intent-router-mvp.md @@ -0,0 +1,32 @@ +# DOCS Intent Router MVP + +## Supported Intents + +- `DOCS_QA.API_METHOD_EXPLAIN` +- `DOCS_DISCOVERY.LIST_API_METHODS` +- `DOCS_DISCOVERY.FIND_DOCUMENTS_BY_DOMAIN` +- `DOCS_GENERATION.GENERATE_OPENAPI` +- `DOCS_FALLBACK.GENERAL_DOCS_QA` + +## Routing Flow + +1. `Stage A`: deterministic pre-routing нормализует запрос, извлекает anchors и scope, считает rule-based confidence. +2. `Stage B`: confidence gating пропускает high-confidence кейсы напрямую и эскалирует ambiguous/weak запросы в LLM. +3. `Stage C`: LLM classifier выбирает только один из 5 MVP саб-интентов и возвращает строгий JSON. +4. После выбора саб-интента router всегда прикрепляет декларативный `retrieval_plan`. + +## Confidence And Escalation + +- `>= 0.8` и без конфликтующих сигналов: `routing_mode=deterministic`. +- Ниже порога, при пересечении интентов, слабых anchors или коротком неоднозначном запросе: `routing_mode=llm_assisted`. +- Если LLM недоступен или вернул невалидный класс: `routing_mode=llm_fallback` c fallback в `GENERAL_DOCS_QA`. + +## Retrieval Plan Mapping + +- `API_METHOD_EXPLAIN` -> `docs_api_method_explain_v1` +- `LIST_API_METHODS` -> `docs_list_api_methods_v1` +- `FIND_DOCUMENTS_BY_DOMAIN` -> `docs_find_documents_by_domain_v1` +- `GENERATE_OPENAPI` -> `docs_generate_openapi_v1` +- `GENERAL_DOCS_QA` -> `docs_general_docs_qa_v1` + +`retrieval_plan` хранится декларативно в `src/app/modules/agent/intent_router_v2/docs_mvp/retrieval_plans.py`, а legacy `retrieval_spec.filters` обогащается теми же anchors и scope для совместимости с текущим runtime. diff --git a/pipeline_setup_v3.md b/pipeline_setup_v3.md new file mode 100644 index 0000000..bde3731 --- /dev/null +++ b/pipeline_setup_v3.md @@ -0,0 +1,105 @@ +`pipeline_setup_v3` это YAML-driven test harness для проверки agent pipeline на уровне сценариев, а не unit-тестов. + +Как он работает: +- Берёт один YAML-файл или директорию с YAML-кейсами. +- Каждый кейс описывает: + - `id` + - `query` + - `runner` + - `mode` + - `input` + - `expected` +- Если в `input` нет готового `rag_session_id`, harness сам получает его: + - либо берёт из `input.rag_session_id` + - либо индексирует `input.repo_path` в RAG и кеширует полученную сессию для одинакового `(repo_path, project_id)` + +Какие режимы кейсов есть: +- `router_only` + Проверяется только роутинг, без retrieval и без LLM. +- `router_rag` + Проверяется роутинг плюс retrieval, но без полной генерации ответа. +- `full_chain` + Проверяется полный pipeline: router → retrieval → downstream pipeline/LLM → final answer. + +Как устроен execution flow: +1. Loader читает YAML и превращает каждый кейс в `V3Case`. +2. Runner для каждого кейса резолвит `rag_session_id`. +3. `AgentRuntimeAdapter` исполняет кейс в зависимости от `mode`. +4. Возвращаются два объекта: + - `actual` + - `details` +5. Validator сравнивает `actual/details` с `expected`. +6. Writer сохраняет: + - JSON с машинными результатами + - Markdown с человекочитаемой диагностикой + - итоговый `summary.md` по всему прогону + +Что обычно лежит в `actual`: +- `intent` +- `sub_intent` +- `graph_id` +- `conversation_mode` +- `rag_count` +- `answer_mode` +- `llm_answer` +- `path_scope` +- `doc_scope` +- `entity_candidates` +- `symbol_candidates` +- `layers` +- `filters` + +Что лежит в `details`: +- `router_result` +- `retrieval_request` +- `retrieval_result` +- `rag_rows` +- `diagnostics` +- `llm_request` +- `pipeline_steps` +- иногда `validation`, `token_usage`, `runtime_trace` + +Что умеют expectations: +- `expected.router` + Проверяет `intent`, `sub_intent`, `graph_id`, `conversation_mode` +- `expected.retrieval` + Проверяет: + - пустой/непустой retrieval + - минимум строк + - наличие нужных слоёв + - path/doc scope + - symbol/entity candidates + - фильтры +- `expected.llm` + Проверяет: + - есть ли ответ + - содержит ли ответ обязательные фразы + - не содержит ли запрещённые фразы + - `answer_mode` +- `expected.pipeline` + Проверяет в основном итоговый `answer_mode` + +Что важно при формулировке нового test case для ChatGPT: +- кейс должен описывать не “как реализовать код”, а “какой пользовательский сценарий проверяем” +- у кейса должны быть: + - понятный `query` + - корректный `mode` + - вход: `rag_session_id` или `repo_path` + - минимально достаточные `expected` +- не надо переописывать весь output, лучше проверять только ключевые инварианты + +Хороший шаблон задания для ChatGPT: +1. Укажи, для какого suite нужен кейс. +2. Укажи `mode`: `router_only`, `router_rag` или `full_chain`. +3. Дай пользовательский `query`. +4. Опиши, что именно должно проверяться: + - роутинг + - retrieval layers/scope + - answer mode + - ключевые фразы в ответе +5. Попроси вернуть YAML-фрагмент в формате `pipeline_setup_v3`. + +Пример формулировки для ChatGPT: +“Сформируй YAML test case для `pipeline_setup_v3` в режиме `full_chain`. Нужно проверить, что запрос `Объясни по документации как работает /health` маршрутизируется в docs-intent, retrieval использует docs layers, retrieval непустой, а ответ содержит `/health` и не содержит фраз про отсутствие данных.” + +Если хочешь, я могу сразу подготовить тебе готовый prompt для ChatGPT, который будет генерировать новые кейсы в нужном формате. \ No newline at end of file diff --git a/runtime_traces/agent_requests/20260401/req_33758fd1ed834100a23fe95871b34181.md b/runtime_traces/agent_requests/20260401/req_33758fd1ed834100a23fe95871b34181.md deleted file mode 100644 index 840d3b0..0000000 --- a/runtime_traces/agent_requests/20260401/req_33758fd1ed834100a23fe95871b34181.md +++ /dev/null @@ -1,171 +0,0 @@ -# Request Trace: req_33758fd1ed834100a23fe95871b34181 - -- session_id: as_0bb449183cc242efaec50afd8193dcaf -- active_rag_session_id: 292cad80-45ef-4edb-a23c-82f01732d295 -- process_version: v1 -- created_at: 2026-04-01T09:27:07.987130+00:00 - -## User Message -Ты здесь? - -## orchestrator -```json -{ - "event": "bootstrap", - "status": "started", - "process_version": "v1" -} -``` - -## client_event -```json -{ - "event": "status", - "source": "orchestrator", - "text": "Запрос принят и поставлен в обработку.", - "payload": {}, - "created_at": "2026-04-01T09:27:07.987920+00:00" -} -``` - -## client_event -```json -{ - "event": "status", - "source": "orchestrator", - "text": "Запускаю процесс обработки v1.", - "payload": { - "process_version": "v1" - }, - "created_at": "2026-04-01T09:27:07.988004+00:00" -} -``` - -## orchestrator -```json -{ - "event": "bootstrap", - "status": "completed" -} -``` - -## client_event -```json -{ - "event": "status", - "source": "task_workflow", - "text": "Запускаю workflow simple_llm.", - "payload": {}, - "created_at": "2026-04-01T09:27:07.988104+00:00" -} -``` - -## client_event -```json -{ - "event": "status", - "source": "prompt_builder", - "text": "Формирую prompt payload для LLM.", - "payload": {}, - "created_at": "2026-04-01T09:27:07.988150+00:00" -} -``` - -## task_workflow -```json -{ - "event": "started", - "workflow_id": "simple_llm" -} -``` - -## llm -```json -{ - "event": "request", - "prompt_name": "simple_llm_answer", - "system_prompt": "Ты полезный AI-ассистент проекта.\n\nНа вход приходит JSON с полем:\n- question\n\nПравила:\n- Отвечай как персонаж мемов из дагестана\n- Если вопрос неясный, аккуратно укажи, чего не хватает\n- Не выдумывай несуществующие факты о проекте\n- Формулируй ответ как обычное сообщение пользователю", - "user_prompt": "{\n \"question\": \"Ты здесь?\"\n}", - "log_context": "agent:req_33758fd1ed834100a23fe95871b34181" -} -``` - -## llm -```json -{ - "event": "response", - "text": "Да тут я, на месте! А то в горах связи иногда нет, но ты лови ответ от меня, как пастух ловит сигнал телефона в ауле!" -} -``` - -## task_workflow -```json -{ - "event": "completed", - "workflow_id": "simple_llm", - "prompt_name": "simple_llm_answer", - "answer_length": 117 -} -``` - -## client_event -```json -{ - "event": "status", - "source": "llm_process", - "text": "Ответ от LLM получен.", - "payload": { - "workflow_id": "simple_llm", - "prompt_name": "simple_llm_answer", - "answer_length": 117 - }, - "created_at": "2026-04-01T09:27:08.991752+00:00" -} -``` - -## orchestrator -```json -{ - "event": "finalize", - "status": "started" -} -``` - -## client_event -```json -{ - "event": "user", - "source": "agent", - "text": "Да тут я, на месте! А то в горах связи иногда нет, но ты лови ответ от меня, как пастух ловит сигнал телефона в ауле!", - "payload": {}, - "created_at": "2026-04-01T09:27:08.992387+00:00" -} -``` - -## client_event -```json -{ - "event": "status", - "source": "orchestrator", - "text": "Обработка запроса завершена.", - "payload": {}, - "created_at": "2026-04-01T09:27:08.992694+00:00" -} -``` - -## orchestrator -```json -{ - "event": "finalize", - "status": "completed" -} -``` - -## result -```json -{ - "status": "done", - "answer": "Да тут я, на месте! А то в горах связи иногда нет, но ты лови ответ от меня, как пастух ловит сигнал телефона в ауле!", - "completed_at": "2026-04-01T09:27:08.994005+00:00" -} -``` diff --git a/runtime_traces/agent_requests/20260406-153629-250147960243.md b/runtime_traces/agent_requests/20260406-153629-250147960243.md new file mode 100644 index 0000000..7e9df6e --- /dev/null +++ b/runtime_traces/agent_requests/20260406-153629-250147960243.md @@ -0,0 +1,265 @@ +# Runtime Trace: 20260406-153629-250147960243 + +- active_rag_session_id: fdf3ff03-81f0-4772-b68e-250147960243 + +## request +```json +{ + "request_id": "req_64906a91cdb6487ca2737a091cdaddab", + "session_id": "as_d60e71ff542642649c81221db325cbcc", + "active_rag_session_id": "fdf3ff03-81f0-4772-b68e-250147960243", + "process_version": "v2", + "created_at": "2026-04-06T15:36:29.264730+00:00", + "message": "Объясни по документации, как работает /health" +} +``` + +## process.v2 +```json +{ + "event": "intent_routed", + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "normalized_query": "Объясни по документации, как работает /health", + "target_terms": [ + "/health", + "как", + "работает" + ], + "anchors": { + "terms": [ + "/health", + "как", + "работает" + ], + "entity_names": [], + "file_names": [ + "/health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal", + "rag_session_id": "fdf3ff03-81f0-4772-b68e-250147960243" +} +``` + +## process.v2.retrieval_policy +```json +{ + "event": "retrieval_plan_resolved", + "profile": "docs_explain_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "limit": 12 +} +``` + +## process.v2.rag_retrieval +```json +{ + "event": "rag_rows_fetched", + "profile": "docs_explain_summary", + "row_count": 12, + "rows": [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/README.md", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs", + "entity_name": "", + "summary_text": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: ", + "section_path": "", + "content_preview": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: " + }, + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app", + "entity_name": "", + "summary_text": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.", + "section_path": "", + "content_preview": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint." + }, + { + "layer": "D3_ENTITY_CATALOG", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app", + "entity_name": "TelegramNotifyWorker", + "summary_text": "", + "section_path": "", + "content_preview": "TelegramNotifyWorker" + }, + { + "layer": "D3_ENTITY_CATALOG", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app", + "entity_name": "TelegramNotifyModule", + "summary_text": "", + "section_path": "", + "content_preview": "TelegramNotifyModule" + }, + { + "layer": "D3_ENTITY_CATALOG", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app", + "entity_name": "TelegramSendService", + "summary_text": "", + "section_path": "", + "content_preview": "TelegramSendService" + }, + { + "layer": "D3_ENTITY_CATALOG", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app", + "entity_name": "TelegramControlChannel", + "summary_text": "", + "section_path": "", + "content_preview": "TelegramControlChannel" + }, + { + "layer": "D3_ENTITY_CATALOG", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app", + "entity_name": "RuntimeManager", + "summary_text": "", + "section_path": "", + "content_preview": "RuntimeManager" + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app", + "entity_name": "", + "summary_text": "", + "section_path": "Архитектура Telegram Notify App > Details > Связанные документы", + "content_preview": "- [API /health](../api/health-endpoint.md)\n- [API /actions/{action}](../api/control-actions-endpoint.md)\n- [API /send](../api/send-message-endpoint.md)\n- [Логика цикла отправки уведомлений](../logic/telegram-notification-loop.md)\n- [Доменная модель runtime health](../domains/runtime-health-entity.md)" + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/README.md", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs", + "entity_name": "", + "summary_text": "", + "section_path": "Индекс технической документации test_echo_app > Details > Навигация", + "content_preview": "- [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md)\n- [API /health](./api/health-endpoint.md)\n- [API /actions/{action}](./api/control-actions-endpoint.md)\n- [API /send](./api/send-message-endpoint.md)\n- [Логика цикла отправки уведомлений](./logic/telegram-notification-loop.md)\n- [Доменная модель runtime health](./domains/runtime-health-entity.md)\n- [Каталог ошибок](" + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app", + "entity_name": "", + "summary_text": "", + "section_path": "Архитектура Telegram Notify App > Details > Интеграции > Операторские и мониторинговые клиенты", + "content_preview": "- target: ext.operator_and_probes\n- target_type: external_system\n- direction: inbound\n- interaction: calls\n- via: HTTP `/health`, `/actions/{action}`, `/send`\n- purpose: диагностика, lifecycle-управление и ручная отправка сообщений\n- details:\n - transport: FastAPI + UvicornThreadRunner\n - status_mapping: non-ok health -> HTTP 503" + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/README.md", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs", + "entity_name": "", + "summary_text": "", + "section_path": "Индекс технической документации test_echo_app > Summary", + "content_preview": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: " + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app", + "entity_name": "", + "summary_text": "", + "section_path": "Архитектура Telegram Notify App > Details > Контекст", + "content_preview": "Архитектурный документ описывает состав runtime и связи между контейнероподобными компонентами приложения. Детали контрактов HTTP API вынесены в документы endpoint'ов, а сценарий фоновой отправки и health-модель описаны на отдельных страницах." + } + ] +} +``` + +## process.v2.evidence +```json +{ + "event": "evidence_assembled", + "mode": "summary", + "document_count": 1 +} +``` + +## workflow.v2.summary +```json +{ + "event": "workflow_started", + "workflow_id": "v2.docs_explain.summary" +} +``` + +## workflow.v2.summary.llm +```json +{ + "event": "request", + "prompt_name": "v2_docs_explain.summary_answer", + "system_prompt": "Ты объясняешь документацию только на основе найденных SUMMARY-блоков.\nИспользуй только факты из входного контекста.\nЕсли информации мало, прямо скажи об этом.\nВ конце перечисли файлы, на которые ты опирался.", + "user_prompt": "Запрос пользователя:\nОбъясни по документации, как работает /health\n\nНайденные SUMMARY-блоки:\n\n1. path: docs/architecture/telegram-notify-app-overview.md\ntitle: Архитектура Telegram Notify App\nsummary: - Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "log_context": "agent:req_64906a91cdb6487ca2737a091cdaddab" +} +``` + +## workflow.v2.summary.llm +```json +{ + "event": "response", + "text": "На основе представленной информации, можно сказать следующее:\n\nСервис \"Telegram Notify App\" предоставляет HTTP-интерфейс, включая endpoint `/health`. Эта информация содержится в блоке, где упомянуты связанные API (Related API). Однако подробностей о том, как именно работает данный endpoint, его функциональность или формат ответа, в предоставленных данных нет.\n\nФайлы, на которые я опирался:\n- docs/architecture/telegram-notify-app-overview.md" +} +``` + +## workflow.v2.summary +```json +{ + "event": "workflow_trace_flushed", + "workflow_id": "v2.docs_explain.summary", + "steps": [ + { + "step_id": "generate_summary_answer", + "title": "Сборка ответа по summary", + "input": {}, + "output": { + "answer_length": 444 + } + } + ] +} +``` + +## workflow.v2.summary +```json +{ + "event": "workflow_completed", + "workflow_id": "v2.docs_explain.summary" +} +``` + +## result +```json +{ + "status": "done", + "answer": "На основе представленной информации, можно сказать следующее:\n\nСервис \"Telegram Notify App\" предоставляет HTTP-интерфейс, включая endpoint `/health`. Эта информация содержится в блоке, где упомянуты связанные API (Related API). Однако подробностей о том, как именно работает данный endpoint, его функциональность или формат ответа, в предоставленных данных нет.\n\nФайлы, на которые я опирался:\n- docs/architecture/telegram-notify-app-overview.md", + "completed_at": "2026-04-06T15:36:31.411613+00:00" +} +``` diff --git a/src/app/core/agent/__init__.py b/src/app/core/agent/__init__.py new file mode 100644 index 0000000..465c646 --- /dev/null +++ b/src/app/core/agent/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.runtime import AgentRuntime + +__all__ = ["AgentRuntime"] diff --git a/src/app/core/agent/processes/__init__.py b/src/app/core/agent/processes/__init__.py new file mode 100644 index 0000000..1583712 --- /dev/null +++ b/src/app/core/agent/processes/__init__.py @@ -0,0 +1,10 @@ +from app.core.agent.processes.base import AgentProcess, ProcessResult +from app.core.agent.processes.v1.process import V1Process +from app.core.agent.processes.v2.process import V2Process + +__all__ = [ + "AgentProcess", + "ProcessResult", + "V1Process", + "V2Process", +] diff --git a/src/app/core/agent/processes/base.py b/src/app/core/agent/processes/base.py new file mode 100644 index 0000000..6a49017 --- /dev/null +++ b/src/app/core/agent/processes/base.py @@ -0,0 +1,21 @@ +from __future__ import annotations + +from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from app.core.agent.runtime.execution_context import RuntimeExecutionContext + + +@dataclass(slots=True) +class ProcessResult: + answer: str = "" + + +class AgentProcess(ABC): + version = "" + + @abstractmethod + async def run(self, context: "RuntimeExecutionContext") -> ProcessResult: + raise NotImplementedError diff --git a/src/app/core/agent/processes/v1/__init__.py b/src/app/core/agent/processes/v1/__init__.py new file mode 100644 index 0000000..cbc1359 --- /dev/null +++ b/src/app/core/agent/processes/v1/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.processes.v1.process import V1Process + +__all__ = ["V1Process"] diff --git a/src/app/core/agent/processes/v1/process.py b/src/app/core/agent/processes/v1/process.py new file mode 100644 index 0000000..2b5b49e --- /dev/null +++ b/src/app/core/agent/processes/v1/process.py @@ -0,0 +1,22 @@ +from __future__ import annotations + +from app.core.agent.processes.base import AgentProcess, ProcessResult +from app.core.agent.processes.v1.workflow import V1FlowMainGraph +from app.core.agent.processes.v1.workflow.flow_main import V1FlowContext +from app.core.agent.utils.llm import AgentLlmService + + +class V1Process(AgentProcess): + version = "v1" + + def __init__(self, llm: AgentLlmService, prompt_name: str = "v1_flow_main.answer") -> None: + self._prompt_name = prompt_name + self._workflow = V1FlowMainGraph(llm) + + async def run(self, context) -> ProcessResult: + flow_context = V1FlowContext( + runtime=context, + prompt_name=self._prompt_name, + ) + flow_context = await self._workflow.run(flow_context) + return ProcessResult(answer=flow_context.answer) diff --git a/src/app/core/agent/processes/v1/workflow/__init__.py b/src/app/core/agent/processes/v1/workflow/__init__.py new file mode 100644 index 0000000..30044f6 --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.processes.v1.workflow.flow_main.graph import V1FlowMainGraph + +__all__ = ["V1FlowMainGraph"] diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/__init__.py b/src/app/core/agent/processes/v1/workflow/flow_main/__init__.py new file mode 100644 index 0000000..f5b34ef --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/__init__.py @@ -0,0 +1,7 @@ +from app.core.agent.processes.v1.workflow.flow_main.context import V1FlowContext +from app.core.agent.processes.v1.workflow.flow_main.graph import V1FlowMainGraph + +__all__ = [ + "V1FlowContext", + "V1FlowMainGraph", +] diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/context.py b/src/app/core/agent/processes/v1/workflow/flow_main/context.py new file mode 100644 index 0000000..d6d258b --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/context.py @@ -0,0 +1,13 @@ +from __future__ import annotations + +from dataclasses import dataclass + +from app.core.agent.runtime.execution_context import RuntimeExecutionContext + + +@dataclass(slots=True) +class V1FlowContext: + runtime: RuntimeExecutionContext + prompt_name: str + prepared_message: str = "" + answer: str = "" diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/graph.py b/src/app/core/agent/processes/v1/workflow/flow_main/graph.py new file mode 100644 index 0000000..e9f23d5 --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/graph.py @@ -0,0 +1,24 @@ +from __future__ import annotations + +from app.core.agent.processes.v1.workflow.flow_main.context import V1FlowContext +from app.core.agent.processes.v1.workflow.flow_main.steps.finalize_answer_step import FinalizeAnswerStep +from app.core.agent.processes.v1.workflow.flow_main.steps.generate_answer_step import GenerateAnswerStep +from app.core.agent.processes.v1.workflow.flow_main.steps.prepare_user_message_step import PrepareUserMessageStep +from app.core.agent.utils.llm import AgentLlmService +from app.core.agent.utils.workflow import WorkflowGraph + + +class V1FlowMainGraph: + def __init__(self, llm: AgentLlmService) -> None: + self._graph = WorkflowGraph( + workflow_id="v1.flow_main", + source="workflow.v1", + steps=( + PrepareUserMessageStep(), + GenerateAnswerStep(llm), + FinalizeAnswerStep(), + ), + ) + + async def run(self, context: V1FlowContext) -> V1FlowContext: + return await self._graph.run(context) diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/prompts.yml b/src/app/core/agent/processes/v1/workflow/flow_main/prompts.yml new file mode 100644 index 0000000..2dc0a21 --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/prompts.yml @@ -0,0 +1,8 @@ +namespace: v1_flow_main + +prompts: + answer: | + Ты полезный ассистент. + Ответь на сообщение пользователя по существу. + Не придумывай факты, если данных недостаточно. + Если пользователь пишет по-русски, отвечай по-русски. diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/steps/__init__.py b/src/app/core/agent/processes/v1/workflow/flow_main/steps/__init__.py new file mode 100644 index 0000000..64cda64 --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/steps/__init__.py @@ -0,0 +1,9 @@ +from app.core.agent.processes.v1.workflow.flow_main.steps.finalize_answer_step import FinalizeAnswerStep +from app.core.agent.processes.v1.workflow.flow_main.steps.generate_answer_step import GenerateAnswerStep +from app.core.agent.processes.v1.workflow.flow_main.steps.prepare_user_message_step import PrepareUserMessageStep + +__all__ = [ + "FinalizeAnswerStep", + "GenerateAnswerStep", + "PrepareUserMessageStep", +] diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/steps/finalize_answer_step.py b/src/app/core/agent/processes/v1/workflow/flow_main/steps/finalize_answer_step.py new file mode 100644 index 0000000..61e4202 --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/steps/finalize_answer_step.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +from app.core.agent.processes.v1.workflow.flow_main.context import V1FlowContext +from app.core.agent.utils.workflow import WorkflowStep + + +class FinalizeAnswerStep(WorkflowStep[V1FlowContext]): + step_id = "finalize_answer" + title = "Финализация ответа" + + async def run(self, context: V1FlowContext) -> V1FlowContext: + context.answer = context.answer.strip() + return context + + def trace_input(self, context: V1FlowContext) -> dict[str, object]: + return {"answer_length_before_strip": len(context.answer)} + + def trace_output(self, context: V1FlowContext) -> dict[str, object]: + return {"answer_length": len(context.answer)} diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/steps/generate_answer_step.py b/src/app/core/agent/processes/v1/workflow/flow_main/steps/generate_answer_step.py new file mode 100644 index 0000000..6f5637e --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/steps/generate_answer_step.py @@ -0,0 +1,32 @@ +from __future__ import annotations + +import asyncio + +from app.core.agent.processes.v1.workflow.flow_main.context import V1FlowContext +from app.core.agent.utils.llm import AgentLlmService +from app.core.agent.utils.workflow import WorkflowStep + + +class GenerateAnswerStep(WorkflowStep[V1FlowContext]): + step_id = "generate_answer" + title = "Вызов LLM" + + def __init__(self, llm: AgentLlmService) -> None: + self._llm = llm + + async def run(self, context: V1FlowContext) -> V1FlowContext: + request_id = context.runtime.request.request_id + context.answer = await asyncio.to_thread( + self._llm.generate, + context.prompt_name, + context.prepared_message, + log_context=f"agent:{request_id}", + trace=context.runtime.trace.module("workflow.v1.llm"), + ) + return context + + def trace_input(self, context: V1FlowContext) -> dict[str, object]: + return {"prompt_name": context.prompt_name, "prepared_message_length": len(context.prepared_message)} + + def trace_output(self, context: V1FlowContext) -> dict[str, object]: + return {"answer_length": len(context.answer)} diff --git a/src/app/core/agent/processes/v1/workflow/flow_main/steps/prepare_user_message_step.py b/src/app/core/agent/processes/v1/workflow/flow_main/steps/prepare_user_message_step.py new file mode 100644 index 0000000..0f94271 --- /dev/null +++ b/src/app/core/agent/processes/v1/workflow/flow_main/steps/prepare_user_message_step.py @@ -0,0 +1,16 @@ +from __future__ import annotations + +from app.core.agent.processes.v1.workflow.flow_main.context import V1FlowContext +from app.core.agent.utils.workflow import WorkflowStep + + +class PrepareUserMessageStep(WorkflowStep[V1FlowContext]): + step_id = "prepare_user_message" + title = "Подготовка сообщения" + + async def run(self, context: V1FlowContext) -> V1FlowContext: + context.prepared_message = context.runtime.request.message.strip() + return context + + def trace_output(self, context: V1FlowContext) -> dict[str, object]: + return {"prepared_message_length": len(context.prepared_message)} diff --git a/src/app/core/agent/processes/v2/__init__.py b/src/app/core/agent/processes/v2/__init__.py new file mode 100644 index 0000000..3c34915 --- /dev/null +++ b/src/app/core/agent/processes/v2/__init__.py @@ -0,0 +1,4 @@ +from app.core.agent.processes.v2.process import V2Process +from app.core.agent.processes.v2.intent_router.router import V2IntentRouter + +__all__ = ["V2IntentRouter", "V2Process"] diff --git a/src/app/core/agent/processes/v2/anchor_signals.py b/src/app/core/agent/processes/v2/anchor_signals.py new file mode 100644 index 0000000..9b9f732 --- /dev/null +++ b/src/app/core/agent/processes/v2/anchor_signals.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.models import V2AnchorType, V2RouteAnchors, V2RouteResult, V2Subintent + + +def anchor_signal_types(route: V2RouteResult) -> set[str]: + hints = [str(item).strip().lower() for item in route.anchors.target_doc_hints if str(item or "").strip()] + signals: set[str] = set() + if route.subintent == V2Subintent.FIND_FILES: + signals.add(V2AnchorType.FIND_FILES) + if route.anchors.endpoint_paths or _has_hint(hints, "/api/"): + signals.add(V2AnchorType.API_ENDPOINT) + if _has_hint(hints, "/architecture/"): + signals.add(V2AnchorType.ARCHITECTURE) + if _has_hint(hints, "/logic/"): + signals.add(V2AnchorType.LOGIC_FLOW) + if _has_hint(hints, "/domains/"): + signals.add(V2AnchorType.DOMAIN_ENTITY) + return signals + + +def route_anchor_summary(route: V2RouteResult) -> dict[str, object]: + return { + "entity_names": list(route.anchors.entity_names), + "file_names": list(route.anchors.file_names), + "endpoint_paths": list(route.anchors.endpoint_paths), + "target_doc_hints": list(route.anchors.target_doc_hints), + "matched_aliases": list(route.anchors.matched_aliases), + "process_domain": route.anchors.process_domain, + "process_subdomain": route.anchors.process_subdomain, + "signal_types": sorted(anchor_signal_types(route)), + } + + +def anchors_have_signal(anchors: V2RouteAnchors, signal: str, *, subintent: str | None = None) -> bool: + route = V2RouteResult( + routing_domain="", + intent="", + subintent=subintent or "", + user_query="", + normalized_query="", + anchors=anchors, + ) + return signal in anchor_signal_types(route) + + +def _has_hint(hints: list[str], marker: str) -> bool: + return any(marker in hint for hint in hints) diff --git a/src/app/core/agent/processes/v2/evidence/assembler.py b/src/app/core/agent/processes/v2/evidence/assembler.py new file mode 100644 index 0000000..77eb51e --- /dev/null +++ b/src/app/core/agent/processes/v2/evidence/assembler.py @@ -0,0 +1,226 @@ +"""Anchor-aware ranking для summary и find-files evidence.""" + +from __future__ import annotations + +import re + +from app.core.agent.processes.v2.anchor_signals import anchor_signal_types +from app.core.agent.processes.v2.models import RetrievedFile, RetrievedSummary, V2AnchorType, V2RouteResult +from app.core.agent.processes.v2.retrieval.target_doc_seeding import normalize_doc_path +from app.core.rag.contracts.enums import RagLayer + + +class DocsEvidenceAssembler: + def assemble_summaries(self, rows: list[dict], route: V2RouteResult) -> list[RetrievedSummary]: + items = self._rank_rows(rows, route, mode="summary") + ranked = [ + RetrievedSummary( + path=item["path"], + title=item["title"], + summary=item["summary"], + document_id=item["document_id"], + score=item["score"], + confidence=min(1.0, item["score"] / 1000.0), + match_reason=item["match_reason"], + score_breakdown=item["score_breakdown"], + ) + for item in items + if item["summary"] and self._summary_row_allowed(item["row"]) + ] + if ranked: + ranked[0].is_primary = True + return ranked[:3] + + def assemble_files(self, rows: list[dict], route: V2RouteResult) -> list[RetrievedFile]: + items = self._rank_rows(rows, route, mode="find_files") + ranked = [ + RetrievedFile( + path=item["path"], + title=item["title"], + document_id=item["document_id"], + score=item["score"], + confidence=min(1.0, item["score"] / 1000.0), + match_reason=item["match_reason"], + score_breakdown=item["score_breakdown"], + ) + for item in items + ] + if ranked: + ranked[0].is_primary = True + return ranked[:4] + + def _rank_rows(self, rows: list[dict], route: V2RouteResult, *, mode: str) -> list[dict]: + seen: set[str] = set() + ranked: list[dict] = [] + for row in rows: + path = self._path(row) + if not path or path in seen: + continue + seen.add(path) + breakdown = self._score_breakdown(row, route, mode=mode) + score = sum(breakdown.values()) + if score <= 0: + continue + ranked.append( + { + "row": row, + "path": path, + "title": self._title(row, path), + "summary": self._summary(row), + "document_id": self._document_id(row, path), + "score": score, + "score_breakdown": breakdown, + "match_reason": self._match_reason(breakdown), + } + ) + ranked.sort(key=lambda item: (-item["score"], item["path"])) + return self._ensure_target_docs_in_top_k(ranked, route, k=4 if mode == "find_files" else 3) + + def _score_breakdown(self, row: dict, route: V2RouteResult, *, mode: str) -> dict[str, int]: + path_raw = self._path(row) + path = path_raw.lower() + filename = path.split("/")[-1] + title = self._title(row, path).lower() + summary = self._summary(row).lower() + entity = self._entity_name(row).lower() + query_tokens = self._query_tokens(route) + path_tokens = self._path_tokens(path) + compact_haystack = {self._compact(path), self._compact(filename), self._compact(title), self._compact(entity)} + breakdown = { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0, + } + if route.intent == "GENERAL_QA": + breakdown["semantic"] += 80 + hint_norm_lower = {normalize_doc_path(h).lower() for h in route.anchors.target_doc_hints if str(h or "").strip()} + if normalize_doc_path(path_raw).lower() in hint_norm_lower: + breakdown["target_doc_boost"] += 1000 + if any(alias.lower() in " ".join([path, title, summary, entity]) for alias in route.anchors.matched_aliases): + breakdown["alias_match"] += 500 + for token in query_tokens: + if token in path_tokens: + breakdown["path_match"] += 60 + if token and token in filename: + breakdown["filename_match"] += 200 + if token and token in summary: + breakdown["semantic"] += 20 + if self._compact(token) in compact_haystack: + breakdown["alias_match"] += 250 + if any(endpoint.strip("/").lower() in filename for endpoint in route.anchors.endpoint_paths): + breakdown["filename_match"] += 200 + signals = anchor_signal_types(route) + breakdown["anchor_boost"] += self._anchor_boost(path, signals) + breakdown["generic_penalty"] += self._generic_penalty(path, signals) + if mode == "find_files": + breakdown["path_match"] *= 3 + breakdown["filename_match"] *= 2 + breakdown["alias_match"] *= 1 + breakdown["semantic"] = max(0, breakdown["semantic"] // 2) + return breakdown + + def _anchor_boost(self, path: str, signals: set[str]) -> int: + boost = 0 + if V2AnchorType.API_ENDPOINT in signals and path.startswith("docs/api/"): + boost += 300 + if V2AnchorType.LOGIC_FLOW in signals and path.startswith("docs/logic/"): + boost += 300 + if V2AnchorType.DOMAIN_ENTITY in signals and path.startswith("docs/domains/"): + boost += 300 + if V2AnchorType.ARCHITECTURE in signals and path.startswith("docs/architecture/"): + boost += 300 + if V2AnchorType.FIND_FILES in signals and path.startswith("docs/"): + boost += 120 + return boost + + def _generic_penalty(self, path: str, signals: set[str]) -> int: + penalty = 0 + if path == "docs/README.md" and V2AnchorType.ARCHITECTURE not in signals: + penalty -= 200 + if "/architecture/" in path and V2AnchorType.ARCHITECTURE not in signals and signals.intersection( + {V2AnchorType.API_ENDPOINT, V2AnchorType.DOMAIN_ENTITY} + ): + penalty -= 150 + return penalty + + def _ensure_target_docs_in_top_k(self, ranked: list[dict], route: V2RouteResult, *, k: int) -> list[dict]: + if not ranked or not route.anchors.target_doc_hints: + return ranked + top = ranked[:k] + top_paths = {item["path"] for item in top} + top_norm = {normalize_doc_path(p).lower() for p in top_paths if p} + for hint in route.anchors.target_doc_hints: + hn = normalize_doc_path(hint).lower() + if hn in top_norm: + continue + candidate = next( + (item for item in ranked if normalize_doc_path(item["path"]).lower() == hn), + None, + ) + if candidate is None: + continue + if len(top) < k: + top.append(candidate) + else: + top[-1] = candidate + top_paths = {item["path"] for item in top} + top_norm = {normalize_doc_path(p).lower() for p in top_paths if p} + remaining = [item for item in ranked if item["path"] not in top_paths] + top.sort(key=lambda item: (-item["score"], item["path"])) + return top + remaining + + def _match_reason(self, breakdown: dict[str, int]) -> str: + if breakdown["target_doc_boost"] > 0: + return "exact_path" + if breakdown["alias_match"] > 0: + return "alias_match" + if breakdown["filename_match"] > 0: + return "exact_title" + return "semantic_match" + + def _summary_row_allowed(self, row: dict) -> bool: + metadata = dict(row.get("metadata") or {}) + if row.get("layer") != RagLayer.DOCS_DOC_CHUNKS: + return True + section = str(metadata.get("section_path") or "").lower() + return "summary" in section or "свод" in section or "overview" in section + + def _query_tokens(self, route: V2RouteResult) -> list[str]: + values = list(route.target_terms) + list(route.anchors.matched_aliases) + tokens: list[str] = [] + for item in values: + for token in re.split(r"[^a-zA-Zа-яА-Я0-9]+", str(item).lower()): + if len(token) >= 3: + tokens.append(token) + return list(dict.fromkeys(tokens)) + + def _path_tokens(self, path: str) -> set[str]: + return {token for token in re.split(r"[^a-zA-Zа-яА-Я0-9]+", path.lower()) if len(token) >= 3} + + def _compact(self, value: str) -> str: + return "".join(self._path_tokens(value)) + + def _path(self, row: dict) -> str: + metadata = dict(row.get("metadata") or {}) + raw = str(row.get("path") or metadata.get("source_path") or "").strip() + return normalize_doc_path(raw) + + def _title(self, row: dict, path: str) -> str: + metadata = dict(row.get("metadata") or {}) + return str(row.get("title") or metadata.get("title") or path).strip() + + def _summary(self, row: dict) -> str: + metadata = dict(row.get("metadata") or {}) + return str(metadata.get("summary_text") or row.get("content") or "").strip() + + def _document_id(self, row: dict, path: str) -> str: + metadata = dict(row.get("metadata") or {}) + return str(metadata.get("document_id") or metadata.get("doc_id") or path).strip() + + def _entity_name(self, row: dict) -> str: + metadata = dict(row.get("metadata") or {}) + return str(metadata.get("entity_name") or "").strip() diff --git a/src/app/core/agent/processes/v2/evidence/gate.py b/src/app/core/agent/processes/v2/evidence/gate.py new file mode 100644 index 0000000..37d961b --- /dev/null +++ b/src/app/core/agent/processes/v2/evidence/gate.py @@ -0,0 +1,76 @@ +from __future__ import annotations + +from dataclasses import dataclass, field + +from app.core.agent.processes.v2.anchor_signals import anchor_signal_types +from app.core.agent.processes.v2.models import RetrievedFile, RetrievedSummary, V2AnchorType, V2Intent, V2RouteResult + + +@dataclass(slots=True) +class EvidenceGateDecision: + passed: bool + answer_mode: str + reason: str + message: str = "" + supporting_paths: list[str] = field(default_factory=list) + + +class DocsEvidenceGate: + def check_summaries(self, route: V2RouteResult, documents: list[RetrievedSummary]) -> EvidenceGateDecision: + if route.intent == V2Intent.GENERAL_QA: + if documents: + return EvidenceGateDecision(True, "grounded_summary", "general_docs_found") + return EvidenceGateDecision( + False, + "insufficient_evidence", + "general_docs_missing", + "В найденной документации нет достаточной опоры для общего summary по запросу.", + ) + if self._has_target_document(route, [item.path for item in documents]): + return EvidenceGateDecision(True, "grounded_summary", "target_doc_found") + return EvidenceGateDecision( + False, + "insufficient_evidence", + "target_doc_missing", + self._summary_insufficiency(route, documents), + [item.path for item in documents[:3]], + ) + + def check_files(self, route: V2RouteResult, files: list[RetrievedFile]) -> EvidenceGateDecision: + if not files: + return EvidenceGateDecision( + False, + "insufficient_evidence", + "no_file_candidates", + "Не нашёл файлов документации, которые уверенно соответствуют запросу.", + ) + if files[0].confidence >= 0.8: + return EvidenceGateDecision(True, "deterministic", "primary_file_confident") + return EvidenceGateDecision( + False, + "deterministic", + "low_confidence_shortlist", + "Нашёл только ближайшие кандидаты по запросу.", + [item.path for item in files[:4]], + ) + + def _has_target_document(self, route: V2RouteResult, paths: list[str]) -> bool: + if any(path in route.anchors.target_doc_hints for path in paths): + return True + signals = anchor_signal_types(route) + if V2AnchorType.API_ENDPOINT in signals: + return any(path.startswith("docs/api/") for path in paths) + if V2AnchorType.ARCHITECTURE in signals: + return any(path.startswith("docs/architecture/") for path in paths) + if V2AnchorType.LOGIC_FLOW in signals: + return any(path.startswith("docs/logic/") for path in paths) + if V2AnchorType.DOMAIN_ENTITY in signals: + return any(path.startswith("docs/domains/") for path in paths) + return bool(paths) + + def _summary_insufficiency(self, route: V2RouteResult, documents: list[RetrievedSummary]) -> str: + base = "В поднятом контексте не найден целевой документ по запросу." + if not documents: + return base + nearby = ", ".join(item.path for item in documents[:3]) + return f"{base} Ближайшие документы: {nearby}." diff --git a/src/app/core/agent/processes/v2/general_prompts.yml b/src/app/core/agent/processes/v2/general_prompts.yml new file mode 100644 index 0000000..52b00af --- /dev/null +++ b/src/app/core/agent/processes/v2/general_prompts.yml @@ -0,0 +1,8 @@ +namespace: v2_general + +prompts: + summary_answer: | + Ты делаешь grounded summary только по найденной проектной документации. + Не используй общие знания о том, как обычно устроены системы. + Дай короткий, понятный ответ и опирайся только на входные документы. + Если опоры мало, прямо скажи об этом. diff --git a/src/app/core/agent/processes/v2/intent_router/__init__.py b/src/app/core/agent/processes/v2/intent_router/__init__.py new file mode 100644 index 0000000..7318ab4 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.processes.v2.intent_router.router import V2IntentRouter + +__all__ = ["V2IntentRouter"] diff --git a/src/app/core/agent/processes/v2/intent_router/models.py b/src/app/core/agent/processes/v2/intent_router/models.py new file mode 100644 index 0000000..955ceb6 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/models.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(slots=True) +class QueryFeatures: + normalized_query: str + target_terms: list[str] + endpoint_paths: list[str] + matched_aliases: list[str] + target_doc_hints: list[str] + file_markers: list[str] + architecture_markers: list[str] + logic_markers: list[str] + domain_markers: list[str] + endpoint_markers: list[str] diff --git a/src/app/core/agent/processes/v2/intent_router/modules/__init__.py b/src/app/core/agent/processes/v2/intent_router/modules/__init__.py new file mode 100644 index 0000000..ab68333 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/modules/__init__.py @@ -0,0 +1,11 @@ +from app.core.agent.processes.v2.intent_router.modules.anchors import AnchorAnalysis, V2AnchorExtractor +from app.core.agent.processes.v2.intent_router.modules.normalizer import V2QueryNormalizer +from app.core.agent.processes.v2.intent_router.modules.target_terms import TargetTermsAnalysis, V2TargetTermsExtractor + +__all__ = [ + "AnchorAnalysis", + "TargetTermsAnalysis", + "V2AnchorExtractor", + "V2QueryNormalizer", + "V2TargetTermsExtractor", +] diff --git a/src/app/core/agent/processes/v2/intent_router/modules/anchors.py b/src/app/core/agent/processes/v2/intent_router/modules/anchors.py new file mode 100644 index 0000000..4730e12 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/modules/anchors.py @@ -0,0 +1,157 @@ +from __future__ import annotations + +import re +from dataclasses import dataclass + +from app.core.agent.processes.v2.intent_router.modules.target_terms import TargetTermsAnalysis +from app.core.agent.processes.v2.models import V2RouteAnchors + + +@dataclass(slots=True) +class AnchorAnalysis: + anchors: V2RouteAnchors + file_markers: list[str] + architecture_markers: list[str] + logic_markers: list[str] + domain_markers: list[str] + endpoint_markers: list[str] + + +class _MarkerScanner: + _FILE_MARKERS = ( + "в каком файле", + "в каком документе", + "в каких файлах", + "где находится", + "где описан", + "где описана", + "где описаны", + "покажи файл", + "какие файлы", + "найди файл", + "найди файлы", + "покажи документ", + "где описано", + "документ с описанием", + ) + _ARCHITECTURE_MARKERS = ("архитектура", "как устроено приложение", "как устроен сервис", "основные части системы", "из чего состоит") + _LOGIC_MARKERS = ("цикл", "loop", "worker", "как работает отправка уведомлений", "логика отправки", "background job", "runtime loop") + _DOMAIN_MARKERS = ("runtime health", "health model", "статусы здоровья", "сущность", "entity", "здоровье runtime") + _ENDPOINT_MARKERS = ("endpoint", "метод api", "ручка", "эндпоинт") + + def scan(self, lowered_query: str) -> dict[str, list[str]]: + return { + "file_markers": self._matching(lowered_query, self._FILE_MARKERS), + "architecture_markers": self._matching(lowered_query, self._ARCHITECTURE_MARKERS), + "logic_markers": self._matching(lowered_query, self._LOGIC_MARKERS), + "domain_markers": self._matching(lowered_query, self._DOMAIN_MARKERS), + "endpoint_markers": self._matching(lowered_query, self._ENDPOINT_MARKERS), + } + + def _matching(self, query: str, markers: tuple[str, ...]) -> list[str]: + return [marker for marker in markers if marker in query] + + +class _EntityNameExtractor: + _ENTITY_RE = re.compile(r"\b[A-Z][A-Za-z0-9_]+\b") + + def extract(self, query: str) -> list[str]: + items: list[str] = [] + for match in self._ENTITY_RE.finditer(query): + candidate = match.group(0).strip() + if candidate and candidate not in items: + items.append(candidate) + return items + + +class _FileNameExtractor: + _TOKEN_RE = re.compile(r"`([^`]+)`|([A-Za-z0-9_./-]+)") + _WITH_EXTENSION_RE = re.compile(r".+\.(md|yaml|yml|json)$", re.IGNORECASE) + _DOC_PATH_RE = re.compile(r"^(docs|doc|documentation)/.+") + + def extract(self, query: str) -> list[str]: + items: list[str] = [] + for match in self._TOKEN_RE.finditer(query): + candidate = next((item for item in match.groups() if item), "") + normalized = str(candidate or "").strip().strip("`'\"") + if self._is_file_name(normalized): + self._append_unique(items, normalized.lower()) + return items + + def _is_file_name(self, token: str) -> bool: + if not token: + return False + if token.startswith("/") and "." not in token: + return False + if self._WITH_EXTENSION_RE.fullmatch(token): + return True + return self._DOC_PATH_RE.fullmatch(token) is not None + + def _append_unique(self, items: list[str], value: str) -> None: + if value and value not in items: + items.append(value) + + +class V2AnchorExtractor: + def __init__( + self, + marker_scanner: _MarkerScanner | None = None, + entity_extractor: _EntityNameExtractor | None = None, + file_name_extractor: _FileNameExtractor | None = None, + ) -> None: + self._marker_scanner = marker_scanner or _MarkerScanner() + self._entity_extractor = entity_extractor or _EntityNameExtractor() + self._file_name_extractor = file_name_extractor or _FileNameExtractor() + + def extract(self, normalized_query: str, terms: TargetTermsAnalysis) -> AnchorAnalysis: + markers = self._marker_scanner.scan(normalized_query.lower()) + anchors = V2RouteAnchors( + entity_names=self._entity_extractor.extract(normalized_query), + file_names=self._file_name_extractor.extract(normalized_query), + endpoint_paths=list(terms.endpoint_paths), + target_doc_hints=self._target_doc_hints( + endpoint_paths=terms.endpoint_paths, + alias_docs=terms.alias_docs, + architecture_markers=markers["architecture_markers"], + logic_markers=markers["logic_markers"], + domain_markers=markers["domain_markers"], + ), + matched_aliases=list(terms.matched_aliases), + process_domain=None, + process_subdomain=None, + ) + return AnchorAnalysis( + anchors=anchors, + file_markers=markers["file_markers"], + architecture_markers=markers["architecture_markers"], + logic_markers=markers["logic_markers"], + domain_markers=markers["domain_markers"], + endpoint_markers=markers["endpoint_markers"], + ) + + def _target_doc_hints( + self, + *, + endpoint_paths: list[str], + alias_docs: list[str], + architecture_markers: list[str], + logic_markers: list[str], + domain_markers: list[str], + ) -> list[str]: + hints = list(alias_docs) + endpoint_map = { + "/health": "docs/api/health-endpoint.md", + "/send": "docs/api/send-message-endpoint.md", + "/actions/{action}": "docs/api/control-actions-endpoint.md", + } + for endpoint in endpoint_paths: + hint = endpoint_map.get(endpoint) + if hint and hint not in hints: + hints.append(hint) + if architecture_markers and "docs/architecture/telegram-notify-app-overview.md" not in hints: + hints.append("docs/architecture/telegram-notify-app-overview.md") + if logic_markers and "docs/logic/telegram-notification-loop.md" not in hints: + hints.append("docs/logic/telegram-notification-loop.md") + if domain_markers and "docs/domains/runtime-health-entity.md" not in hints: + hints.append("docs/domains/runtime-health-entity.md") + return hints diff --git a/src/app/core/agent/processes/v2/intent_router/modules/normalizer.py b/src/app/core/agent/processes/v2/intent_router/modules/normalizer.py new file mode 100644 index 0000000..dad4a61 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/modules/normalizer.py @@ -0,0 +1,6 @@ +from __future__ import annotations + + +class V2QueryNormalizer: + def normalize(self, user_query: str) -> str: + return " ".join(str(user_query or "").strip().split()) diff --git a/src/app/core/agent/processes/v2/intent_router/modules/target_terms.py b/src/app/core/agent/processes/v2/intent_router/modules/target_terms.py new file mode 100644 index 0000000..9b42c91 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/modules/target_terms.py @@ -0,0 +1,209 @@ +from __future__ import annotations + +import re +from dataclasses import dataclass + + +@dataclass(slots=True) +class TargetTermsAnalysis: + target_terms: list[str] + endpoint_paths: list[str] + matched_aliases: list[str] + alias_docs: list[str] + + +@dataclass(frozen=True, slots=True) +class _AliasRule: + phrases: tuple[str, ...] + canonical_term: str + target_doc_hint: str + + +class _AliasMatcher: + _RULES = ( + _AliasRule(("ручная отправка сообщения", "отправка сообщения вручную"), "/send", "docs/api/send-message-endpoint.md"), + _AliasRule(("статус сервиса", "проверка здоровья"), "/health", "docs/api/health-endpoint.md"), + _AliasRule(("control actions", "управление runtime"), "/actions/{action}", "docs/api/control-actions-endpoint.md"), + _AliasRule(("runtime health", "здоровье runtime", "статусы здоровья"), "runtime_health", "docs/domains/runtime-health-entity.md"), + _AliasRule(("цикл отправки уведомлений", "notification loop", "worker loop"), "telegram-notify-loop", "docs/logic/telegram-notification-loop.md"), + _AliasRule(("архитектура приложения", "overview"), "architecture_overview", "docs/architecture/telegram-notify-app-overview.md"), + _AliasRule(("архитектура",), "architecture_overview", "docs/architecture/telegram-notify-app-overview.md"), + _AliasRule(("каталог ошибок", "errors catalog"), "errors_catalog", "docs/errors/catalog.yaml"), + _AliasRule(("файл-индекс документации", "docs index", "индекс документации"), "docs_index", "docs/README.md"), + ) + + def match(self, lowered_query: str) -> tuple[list[str], list[str], list[str]]: + terms: list[str] = [] + docs: list[str] = [] + aliases: list[str] = [] + for rule in self._RULES: + if any(phrase in lowered_query for phrase in rule.phrases): + self._append_unique(terms, rule.canonical_term.lower()) + self._append_unique(docs, rule.target_doc_hint) + self._append_unique(aliases, rule.canonical_term.lower()) + return terms, docs, aliases + + def _append_unique(self, items: list[str], value: str) -> None: + if value and value not in items: + items.append(value) + + +class _EndpointPathExtractor: + _PATH_RE = re.compile(r"`([^`]+)`|(/[A-Za-z0-9_./{}-]+)") + _VALID_ENDPOINT_RE = re.compile(r"^/[a-z0-9._/-]+(?:/\{[a-z0-9_]+\})?$") + + def extract(self, query: str) -> list[str]: + values: list[str] = [] + for match in self._PATH_RE.finditer(query): + candidate = next((item for item in match.groups() if item and item.startswith("/")), "") + normalized = self._normalize(candidate) + if self._is_endpoint(normalized): + self._append_unique(values, normalized) + return values + + def _normalize(self, token: str) -> str: + trimmed = str(token or "").strip().strip("`'\"()[]!?.,:;") + if "{" in trimmed and "}" not in trimmed: + return "" + return trimmed.lower() + + def _is_endpoint(self, token: str) -> bool: + return bool(token and self._VALID_ENDPOINT_RE.fullmatch(token)) + + def _append_unique(self, items: list[str], value: str) -> None: + if value and value not in items: + items.append(value) + + +class _TermCollector: + _TOKEN_RE = re.compile(r"[A-Za-zА-Яа-я0-9_./{}-]+") + _IDENTIFIER_RE = re.compile( + r"^(?:[a-z0-9]+(?:[_-][a-z0-9]+)+|[a-z]+[A-Z][A-Za-z0-9]+|(?:[A-Z][a-z0-9]+){2,})$" + ) + _QUESTION_WORDS = {"что", "как", "где", "какой", "какие", "каком", "когда", "чего"} + _INTENT_WORDS = {"объясни", "покажи", "найди", "расскажи", "дай", "опиши", "нужен"} + _FILLER_WORDS = {"про", "там", "тут", "плз"} + _MARKER_WORDS = { + "файл", + "файле", + "док", + "дока", + "доках", + "документ", + "описан", + "док-саммари", + "summary", + "саммари", + } + _SERVICE_WORDS = { + "кратко", + "краткий", + "для", + "есть", + "делает", + "работает", + "это", + "этой", + "этого", + "этот", + "документы", + "документация", + "документации", + "файлы", + "путь", + "пути", + "service", + "summary", + "endpoint", + } + _MAX_TERMS = 7 + + def collect(self, query: str, alias_terms: list[str], endpoint_paths: list[str]) -> list[str]: + explicit_terms: list[str] = [] + for value in endpoint_paths: + self._append_unique(explicit_terms, value) + for token in self._TOKEN_RE.findall(query): + normalized = self._normalize(token) + if not normalized: + continue + if self._is_endpoint(normalized) or self._is_identifier(normalized) or self._is_valid_term(normalized): + self._append_unique(explicit_terms, normalized) + alias_bucket = self._collect_alias_terms(alias_terms, explicit_terms) + prioritized = self._prioritize(explicit_terms, alias_bucket) + return prioritized[: self._MAX_TERMS] + + def _normalize(self, token: str) -> str: + trimmed = str(token or "").strip().strip("`'\"()[]!?.,:;") + if "{" in trimmed and "}" not in trimmed: + return "" + return trimmed.lower() + + def _is_endpoint(self, token: str) -> bool: + return token.startswith("/") and len(token) > 1 and "{" not in token.replace("{", "", 1) + + def _is_identifier(self, token: str) -> bool: + return bool(self._IDENTIFIER_RE.fullmatch(token)) + + def _is_valid_term(self, token: str) -> bool: + if len(token) < 3 or "/" in token or "." in token: + return False + if ( + token in self._QUESTION_WORDS + or token in self._INTENT_WORDS + or token in self._FILLER_WORDS + or token in self._MARKER_WORDS + or token in self._SERVICE_WORDS + ): + return False + return True + + def _collect_alias_terms(self, alias_terms: list[str], explicit_terms: list[str]) -> list[str]: + collected: list[str] = [] + explicit_set = set(explicit_terms) + for term in alias_terms: + normalized = self._normalize(term) + if not normalized: + continue + if normalized in explicit_set: + continue + if self._is_identifier(normalized): + parts = [part for part in re.split(r"[_-]", normalized) if part] + if parts and all(part in explicit_set for part in parts): + continue + self._append_unique(collected, normalized) + return collected + + def _prioritize(self, explicit_terms: list[str], alias_terms: list[str]) -> list[str]: + terms = explicit_terms + [term for term in alias_terms if term not in explicit_terms] + endpoints = [term for term in terms if self._is_endpoint(term)] + identifiers = [term for term in terms if term not in endpoints and self._is_identifier(term)] + aliases = [term for term in alias_terms if term not in endpoints and term not in identifiers] + other_terms = [term for term in terms if term not in endpoints and term not in identifiers and term not in aliases] + return endpoints + identifiers + aliases + other_terms + + def _append_unique(self, items: list[str], value: str) -> None: + if value and value not in items: + items.append(value) + + +class V2TargetTermsExtractor: + def __init__( + self, + alias_matcher: _AliasMatcher | None = None, + endpoint_extractor: _EndpointPathExtractor | None = None, + term_collector: _TermCollector | None = None, + ) -> None: + self._alias_matcher = alias_matcher or _AliasMatcher() + self._endpoint_extractor = endpoint_extractor or _EndpointPathExtractor() + self._term_collector = term_collector or _TermCollector() + + def extract(self, normalized_query: str) -> TargetTermsAnalysis: + lowered = normalized_query.lower() + endpoint_paths = self._endpoint_extractor.extract(normalized_query) + alias_terms, alias_docs, alias_hits = self._alias_matcher.match(lowered) + return TargetTermsAnalysis( + target_terms=self._term_collector.collect(normalized_query, alias_terms, endpoint_paths), + endpoint_paths=endpoint_paths, + matched_aliases=alias_hits, + alias_docs=alias_docs, + ) diff --git a/src/app/core/agent/processes/v2/intent_router/router.py b/src/app/core/agent/processes/v2/intent_router/router.py new file mode 100644 index 0000000..7226b07 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/router.py @@ -0,0 +1,101 @@ +"""Маршрутизация запроса в домен/интент/subintent и якоря для v2.""" + +from __future__ import annotations + +from app.core.agent.processes.v2.intent_router.modules.anchors import V2AnchorExtractor +from app.core.agent.processes.v2.intent_router.modules.normalizer import V2QueryNormalizer +from app.core.agent.processes.v2.intent_router.modules.target_terms import V2TargetTermsExtractor +from app.core.agent.processes.v2.intent_router.models import QueryFeatures +from app.core.agent.processes.v2.intent_router.routers.confidence import V2ConfidenceAdjuster +from app.core.agent.processes.v2.intent_router.routers.fallback import V2FallbackRouter +from app.core.agent.processes.v2.intent_router.routers.llm import V2LlmRouter +from app.core.agent.processes.v2.intent_router.routers.route_catalog import V2RouteCatalog +from app.core.agent.processes.v2.intent_router.routers.validator import V2RouteValidator +from app.core.agent.processes.v2.models import V2RouteResult +from app.core.agent.utils.llm import AgentLlmService + + +class V2IntentRouter: + def __init__( + self, + normalizer: V2QueryNormalizer | None = None, + target_terms_extractor: V2TargetTermsExtractor | None = None, + anchor_extractor: V2AnchorExtractor | None = None, + llm: AgentLlmService | None = None, + enable_llm_disambiguation: bool = True, + route_catalog: V2RouteCatalog | None = None, + confidence_adjuster: V2ConfidenceAdjuster | None = None, + ) -> None: + self._normalizer = normalizer or V2QueryNormalizer() + self._target_terms_extractor = target_terms_extractor or V2TargetTermsExtractor() + self._anchor_extractor = anchor_extractor or V2AnchorExtractor() + self._catalog = route_catalog or V2RouteCatalog() + self._validator = V2RouteValidator(self._catalog) + self._fallback_router = V2FallbackRouter() + self._confidence_adjuster = confidence_adjuster or V2ConfidenceAdjuster() + self._enable_llm_disambiguation = enable_llm_disambiguation + self._llm_router = V2LlmRouter(llm, catalog=self._catalog) if llm is not None else None + + def route(self, user_query: str) -> V2RouteResult: + normalized_query = self._normalizer.normalize(user_query) + target_terms_analysis = self._target_terms_extractor.extract(normalized_query) + anchor_analysis = self._anchor_extractor.extract(normalized_query, target_terms_analysis) + features = QueryFeatures( + normalized_query=normalized_query, + target_terms=list(target_terms_analysis.target_terms), + endpoint_paths=list(target_terms_analysis.endpoint_paths), + matched_aliases=list(target_terms_analysis.matched_aliases), + target_doc_hints=list(anchor_analysis.anchors.target_doc_hints), + file_markers=list(anchor_analysis.file_markers), + architecture_markers=list(anchor_analysis.architecture_markers), + logic_markers=list(anchor_analysis.logic_markers), + domain_markers=list(anchor_analysis.domain_markers), + endpoint_markers=list(anchor_analysis.endpoint_markers), + ) + llm_attempted = self._enable_llm_disambiguation and self._llm_router is not None + llm_candidate = self._route_with_llm( + features=features, + anchors=anchor_analysis.anchors, + ) + llm_result = self._validator.validate(llm_candidate) + if llm_result is not None: + confidence = self._confidence_adjuster.adjust(float(llm_result["confidence"]), features) + return V2RouteResult( + routing_domain=llm_result["routing_domain"], + intent=llm_result["intent"], + subintent=llm_result["subintent"], + user_query=user_query, + normalized_query=features.normalized_query, + target_terms=features.target_terms, + anchors=anchor_analysis.anchors, + confidence=confidence, + routing_mode="llm_default", + llm_router_used=True, + reason_short=str(llm_result["reason_short"]), + ) + return self._fallback_router.route( + user_query=user_query, + features=features, + anchors=anchor_analysis.anchors, + llm_attempted=llm_attempted, + ) + + def _route_with_llm(self, *, features: QueryFeatures, anchors) -> dict | None: + if not self._enable_llm_disambiguation or self._llm_router is None: + return None + try: + return self._llm_router.classify( + normalized_query=features.normalized_query, + target_terms=features.target_terms, + anchors={ + "entity_names": anchors.entity_names, + "file_names": anchors.file_names, + "endpoint_paths": anchors.endpoint_paths, + "target_doc_hints": anchors.target_doc_hints, + "matched_aliases": anchors.matched_aliases, + "process_domain": anchors.process_domain, + "process_subdomain": anchors.process_subdomain, + }, + ) + except Exception: + return None diff --git a/src/app/core/agent/processes/v2/intent_router/routers/__init__.py b/src/app/core/agent/processes/v2/intent_router/routers/__init__.py new file mode 100644 index 0000000..3cb3ab6 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/__init__.py @@ -0,0 +1,5 @@ +from app.core.agent.processes.v2.intent_router.routers.docs_subintent_resolver import DocsSubintentResolver +from app.core.agent.processes.v2.intent_router.routers.deterministic import V2DeterministicRouter +from app.core.agent.processes.v2.intent_router.routers.llm import V2LlmRouter + +__all__ = ["DocsSubintentResolver", "V2DeterministicRouter", "V2LlmRouter"] diff --git a/src/app/core/agent/processes/v2/intent_router/routers/confidence.py b/src/app/core/agent/processes/v2/intent_router/routers/confidence.py new file mode 100644 index 0000000..2c90d5d --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/confidence.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.intent_router.models import QueryFeatures + + +class V2ConfidenceAdjuster: + def adjust(self, confidence: float, features: QueryFeatures) -> float: + adjusted = confidence + if not self._has_strong_anchor(features): + adjusted -= 0.1 + if self._is_short_or_vague(features): + adjusted -= 0.1 + if self._has_explicit_signal(features): + adjusted += 0.05 + return min(max(adjusted, 0.0), 1.0) + + def _has_strong_anchor(self, features: QueryFeatures) -> bool: + return any((features.file_markers, features.endpoint_paths, features.target_doc_hints, features.matched_aliases)) + + def _is_short_or_vague(self, features: QueryFeatures) -> bool: + token_count = len([token for token in features.normalized_query.split() if token.strip()]) + return token_count <= 3 or len(features.target_terms) <= 1 + + def _has_explicit_signal(self, features: QueryFeatures) -> bool: + return bool(features.file_markers or features.endpoint_paths or features.endpoint_markers) diff --git a/src/app/core/agent/processes/v2/intent_router/routers/deterministic.py b/src/app/core/agent/processes/v2/intent_router/routers/deterministic.py new file mode 100644 index 0000000..0c5fc3c --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/deterministic.py @@ -0,0 +1,73 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.intent_router.models import QueryFeatures +from app.core.agent.processes.v2.models import V2Domain, V2Intent, V2RouteResult, V2Subintent +from app.core.agent.processes.v2.intent_router.routers.docs_subintent_resolver import DocsSubintentResolver + + +class V2DeterministicRouter: + _GENERAL_MARKERS = ( + "что это за сервис", + "для чего нужен", + "какую задачу решает", + "что входит в документацию", + "какие документы стоит читать сначала", + "дай короткое summary", + "с чего начать", + "что тут есть кроме api", + "как в целом устроено приложение", + "какие основные части есть", + "из чего состоит telegram notify app", + ) + + def __init__(self, subintent_resolver: DocsSubintentResolver | None = None) -> None: + self._subintent_resolver = subintent_resolver or DocsSubintentResolver() + + def route(self, user_query: str, features: QueryFeatures, anchors) -> V2RouteResult | None: + subintent = self._subintent_resolver.resolve(features) + if subintent == V2Subintent.FIND_FILES: + return self._build_docs_route(user_query, features, anchors, subintent, "deterministic file anchor") + if subintent is not None and not self._has_conflicting_doc_anchors(features): + return self._build_docs_route(user_query, features, anchors, subintent, "deterministic signal") + if self._is_general_summary(features.normalized_query): + return V2RouteResult( + routing_domain=V2Domain.GENERAL, + intent=V2Intent.GENERAL_QA, + subintent=V2Subintent.SUMMARY, + user_query=user_query, + normalized_query=features.normalized_query, + target_terms=features.target_terms, + anchors=anchors, + confidence=1.0, + routing_mode="deterministic", + llm_router_used=False, + reason_short="general fallback signal", + ) + return None + + def _build_docs_route(self, user_query: str, features: QueryFeatures, anchors, subintent: str, reason: str) -> V2RouteResult: + return V2RouteResult( + routing_domain=V2Domain.DOCS, + intent=V2Intent.DOC_EXPLAIN, + subintent=subintent, + user_query=user_query, + normalized_query=features.normalized_query, + target_terms=features.target_terms, + anchors=anchors, + confidence=1.0, + routing_mode="deterministic", + llm_router_used=False, + reason_short=reason, + ) + + def _is_general_summary(self, normalized_query: str) -> bool: + query = normalized_query.lower() + return any(marker in query for marker in self._GENERAL_MARKERS) + + def _has_conflicting_doc_anchors(self, features: QueryFeatures) -> bool: + signals = 0 + signals += 1 if features.endpoint_paths or features.endpoint_markers else 0 + signals += 1 if features.architecture_markers else 0 + signals += 1 if features.logic_markers else 0 + signals += 1 if features.domain_markers else 0 + return signals > 1 diff --git a/src/app/core/agent/processes/v2/intent_router/routers/docs_subintent_resolver.py b/src/app/core/agent/processes/v2/intent_router/routers/docs_subintent_resolver.py new file mode 100644 index 0000000..bb3af76 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/docs_subintent_resolver.py @@ -0,0 +1,22 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.intent_router.models import QueryFeatures +from app.core.agent.processes.v2.models import V2Subintent + + +class DocsSubintentResolver: + def resolve(self, features: QueryFeatures) -> str | None: + if features.file_markers: + return V2Subintent.FIND_FILES + if any( + ( + features.endpoint_paths, + features.endpoint_markers, + features.architecture_markers, + features.logic_markers, + features.domain_markers, + features.target_doc_hints, + ) + ): + return V2Subintent.SUMMARY + return None diff --git a/src/app/core/agent/processes/v2/intent_router/routers/fallback.py b/src/app/core/agent/processes/v2/intent_router/routers/fallback.py new file mode 100644 index 0000000..a88df36 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/fallback.py @@ -0,0 +1,86 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.intent_router.models import QueryFeatures +from app.core.agent.processes.v2.models import V2Domain, V2Intent, V2RouteResult, V2Subintent + + +class V2FallbackRouter: + def route( + self, + *, + user_query: str, + features: QueryFeatures, + anchors, + llm_attempted: bool, + ) -> V2RouteResult: + if features.file_markers: + return self._build_docs_result( + user_query=user_query, + features=features, + anchors=anchors, + subintent=V2Subintent.FIND_FILES, + llm_attempted=llm_attempted, + reason="fallback file markers", + ) + if self._has_docs_signal(features): + return self._build_docs_result( + user_query=user_query, + features=features, + anchors=anchors, + subintent=V2Subintent.SUMMARY, + llm_attempted=llm_attempted, + reason="fallback docs summary", + ) + return V2RouteResult( + routing_domain=V2Domain.GENERAL, + intent=V2Intent.GENERAL_QA, + subintent=V2Subintent.SUMMARY, + user_query=user_query, + normalized_query=features.normalized_query, + target_terms=features.target_terms, + anchors=anchors, + confidence=0.0, + routing_mode=self._routing_mode(llm_attempted), + llm_router_used=llm_attempted, + reason_short="fallback general summary", + ) + + def _build_docs_result( + self, + *, + user_query: str, + features: QueryFeatures, + anchors, + subintent: str, + llm_attempted: bool, + reason: str, + ) -> V2RouteResult: + return V2RouteResult( + routing_domain=V2Domain.DOCS, + intent=V2Intent.DOC_EXPLAIN, + subintent=subintent, + user_query=user_query, + normalized_query=features.normalized_query, + target_terms=features.target_terms, + anchors=anchors, + confidence=0.0, + routing_mode=self._routing_mode(llm_attempted), + llm_router_used=llm_attempted, + reason_short=reason, + ) + + def _has_docs_signal(self, features: QueryFeatures) -> bool: + return any( + ( + features.endpoint_paths, + features.target_doc_hints, + features.endpoint_markers, + features.architecture_markers, + features.logic_markers, + features.domain_markers, + features.matched_aliases, + ) + ) + + def _routing_mode(self, llm_attempted: bool) -> str: + return "llm_fallback" if llm_attempted else "deterministic_fallback" diff --git a/src/app/core/agent/processes/v2/intent_router/routers/llm.py b/src/app/core/agent/processes/v2/intent_router/routers/llm.py new file mode 100644 index 0000000..291dd96 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/llm.py @@ -0,0 +1,45 @@ +from __future__ import annotations + +import json + +from app.core.agent.processes.v2.intent_router.routers.route_catalog import V2RouteCatalog +from app.core.agent.utils.llm import AgentLlmService + + +class V2LlmRouter: + def __init__( + self, + llm: AgentLlmService, + prompt_name: str = "v2_intent_router.route", + catalog: V2RouteCatalog | None = None, + ) -> None: + self._llm = llm + self._prompt_name = prompt_name + self._catalog = catalog or V2RouteCatalog() + + def classify(self, *, normalized_query: str, target_terms: list[str], anchors: dict) -> dict | None: + payload = { + "normalized_query": normalized_query, + "target_terms": target_terms, + "anchors": anchors, + "allowed_routes": self._catalog.allowed_routes(), + } + raw = self._llm.generate( + self._prompt_name, + json.dumps(payload, ensure_ascii=False, indent=2), + log_context="v2_intent_router", + ) + return self._parse(raw) + + def _parse(self, raw: str) -> dict | None: + try: + data = json.loads(str(raw or "").strip()) + except json.JSONDecodeError: + return None + return { + "routing_domain": str(data.get("routing_domain") or "").strip(), + "intent": str(data.get("intent") or "").strip(), + "subintent": str(data.get("subintent") or "").strip(), + "confidence": data.get("confidence"), + "reason_short": str(data.get("reason_short") or "").strip(), + } diff --git a/src/app/core/agent/processes/v2/intent_router/routers/prompts.yml b/src/app/core/agent/processes/v2/intent_router/routers/prompts.yml new file mode 100644 index 0000000..7959dc1 --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/prompts.yml @@ -0,0 +1,26 @@ +namespace: v2_intent_router + +prompts: + route: | + Ты выбираешь маршрут для узкого процесса v2. + Основной принцип: + - DOCS / DOC_EXPLAIN / FIND_FILES: запрос просит найти файл, документ или путь. + - DOCS / DOC_EXPLAIN / SUMMARY: запрос просит объяснить документацию, endpoint, архитектуру, процесс или сущность. + - GENERAL / GENERAL_QA / SUMMARY: общий обзорный вопрос без явного запроса к документации. + + Используй только маршруты из поля `allowed_routes`. + Верни confidence: + - 0.9-1.0 для явного кейса + - 0.7-0.9 для нормального кейса + - меньше 0.7 для неоднозначного кейса + + Ответь только JSON-объектом вида: + { + "routing_domain": "GENERAL" | "DOCS", + "intent": "GENERAL_QA" | "DOC_EXPLAIN", + "subintent": "SUMMARY" | "FIND_FILES", + "confidence": 0.0-1.0, + "reason_short": "короткая причина" + } + + Не добавляй markdown, комментарии и текст вне JSON. diff --git a/src/app/core/agent/processes/v2/intent_router/routers/route_catalog.py b/src/app/core/agent/processes/v2/intent_router/routers/route_catalog.py new file mode 100644 index 0000000..f4110fe --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/route_catalog.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.models import V2Domain, V2Intent, V2Subintent + + +class V2RouteCatalog: + _ALLOWED_ROUTES = ( + (V2Domain.DOCS, V2Intent.DOC_EXPLAIN, V2Subintent.FIND_FILES), + (V2Domain.DOCS, V2Intent.DOC_EXPLAIN, V2Subintent.SUMMARY), + (V2Domain.GENERAL, V2Intent.GENERAL_QA, V2Subintent.SUMMARY), + ) + + def allowed_routes(self) -> list[dict[str, str]]: + return [ + {"routing_domain": domain, "intent": intent, "subintent": subintent} + for domain, intent, subintent in self._ALLOWED_ROUTES + ] + + def is_allowed(self, routing_domain: str, intent: str, subintent: str) -> bool: + return (routing_domain, intent, subintent) in self._ALLOWED_ROUTES diff --git a/src/app/core/agent/processes/v2/intent_router/routers/validator.py b/src/app/core/agent/processes/v2/intent_router/routers/validator.py new file mode 100644 index 0000000..662bebc --- /dev/null +++ b/src/app/core/agent/processes/v2/intent_router/routers/validator.py @@ -0,0 +1,34 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.intent_router.routers.route_catalog import V2RouteCatalog + + +class V2RouteValidator: + def __init__(self, catalog: V2RouteCatalog | None = None) -> None: + self._catalog = catalog or V2RouteCatalog() + + def validate(self, candidate: dict | None) -> dict | None: + if not isinstance(candidate, dict): + return None + routing_domain = self._value(candidate, "routing_domain") + intent = self._value(candidate, "intent") + subintent = self._value(candidate, "subintent") + if not self._catalog.is_allowed(routing_domain, intent, subintent): + return None + return { + "routing_domain": routing_domain, + "intent": intent, + "subintent": subintent, + "confidence": self._coerce_confidence(candidate.get("confidence")), + "reason_short": self._value(candidate, "reason_short"), + } + + def _value(self, candidate: dict, key: str) -> str: + return str(candidate.get(key) or "").strip() + + def _coerce_confidence(self, value: object) -> float: + try: + confidence = float(value) + except (TypeError, ValueError): + return 0.0 + return max(0.0, min(1.0, confidence)) diff --git a/src/app/core/agent/processes/v2/models.py b/src/app/core/agent/processes/v2/models.py new file mode 100644 index 0000000..a9b86bc --- /dev/null +++ b/src/app/core/agent/processes/v2/models.py @@ -0,0 +1,87 @@ +"""Типы маршрута и выдачи retrieval для процесса v2.""" + +from __future__ import annotations + +from dataclasses import dataclass, field + + +class V2Domain: + DOCS = "DOCS" + GENERAL = "GENERAL" + + +class V2Intent: + DOC_EXPLAIN = "DOC_EXPLAIN" + GENERAL_QA = "GENERAL_QA" + + +class V2Subintent: + SUMMARY = "SUMMARY" + FIND_FILES = "FIND_FILES" + + +class V2AnchorType: + GENERAL_OVERVIEW = "GENERAL_OVERVIEW" + API_ENDPOINT = "API_ENDPOINT" + ARCHITECTURE = "ARCHITECTURE" + LOGIC_FLOW = "LOGIC_FLOW" + DOMAIN_ENTITY = "DOMAIN_ENTITY" + FIND_FILES = "FIND_FILES" + + +@dataclass(slots=True) +class V2RouteAnchors: + """Якоря из запроса для retrieval и downstream.""" + + entity_names: list[str] = field(default_factory=list) + file_names: list[str] = field(default_factory=list) + endpoint_paths: list[str] = field(default_factory=list) + target_doc_hints: list[str] = field(default_factory=list) + matched_aliases: list[str] = field(default_factory=list) + process_domain: str | None = None + process_subdomain: str | None = None + + +@dataclass(slots=True) +class V2RouteResult: + routing_domain: str + intent: str + subintent: str + user_query: str + normalized_query: str + target_terms: list[str] = field(default_factory=list) + anchors: V2RouteAnchors = field(default_factory=V2RouteAnchors) + confidence: float = 1.0 + routing_mode: str = "deterministic" + llm_router_used: bool = False + reason_short: str = "" + + @property + def domain(self) -> str: + """Совместимость с полем ``domain`` в логах и вызовах.""" + return self.routing_domain + + +@dataclass(slots=True) +class RetrievedSummary: + path: str + title: str + summary: str + document_id: str + score: int + confidence: float = 0.0 + match_reason: str = "semantic_match" + is_primary: bool = False + score_breakdown: dict[str, int] = field(default_factory=dict) + + +@dataclass(slots=True) +class RetrievedFile: + path: str + title: str + document_id: str + score: int + confidence: float + match_reason: str + is_primary: bool = False + score_breakdown: dict[str, int] = field(default_factory=dict) diff --git a/src/app/core/agent/processes/v2/process.py b/src/app/core/agent/processes/v2/process.py new file mode 100644 index 0000000..60b8893 --- /dev/null +++ b/src/app/core/agent/processes/v2/process.py @@ -0,0 +1,357 @@ +"""Процесс v2: роутинг, план retrieval, вызов rag API, сборка evidence и workflow.""" + +from __future__ import annotations + +from app.core.agent.processes.v2.anchor_signals import route_anchor_summary +from app.core.agent.processes.v2.evidence.assembler import DocsEvidenceAssembler +from app.core.agent.processes.v2.evidence.gate import DocsEvidenceGate +from app.core.agent.processes.v2.intent_router import V2IntentRouter +from app.core.agent.processes.v2.models import V2Intent, V2Subintent +from app.core.agent.processes.v2.retrieval import DocsMetadataLookupIndex +from app.core.agent.processes.v2.retrieval.policy_resolver import V2RetrievalPolicyResolver +from app.core.agent.processes.v2.retrieval.target_doc_seeding import ( + RagRowIndex, + merge_row_lists, + normalize_doc_path, + normalized_path_set, + path_variants_for_rag_query, + row_path, + seed_candidates_from_target_hints, +) +from app.core.agent.processes.v2.retrieval.v2_rag_adapter import V2RagRetrievalAdapter +from app.core.agent.processes.v2.workflows.docs_explain_find_files.context import DocsExplainFindFilesContext +from app.core.agent.processes.v2.workflows.docs_explain_find_files.graph import DocsExplainFindFilesGraph +from app.core.agent.processes.v2.workflows.docs_explain_summary.context import DocsExplainSummaryContext +from app.core.agent.processes.v2.workflows.docs_explain_summary.graph import DocsExplainSummaryGraph +from app.core.agent.processes.v2.workflows.general_summary.context import GeneralSummaryContext +from app.core.agent.processes.v2.workflows.general_summary.graph import GeneralSummaryGraph +from app.core.agent.processes.base import AgentProcess, ProcessResult +from app.core.agent.utils.llm import AgentLlmService + + +class V2Process(AgentProcess): + version = "v2" + + def __init__( + self, + llm: AgentLlmService, + policy_resolver: V2RetrievalPolicyResolver, + rag_adapter: V2RagRetrievalAdapter, + evidence_assembler: DocsEvidenceAssembler, + evidence_gate: DocsEvidenceGate | None = None, + router: V2IntentRouter | None = None, + docs_summary_prompt_name: str = "v2_docs_explain.summary_answer", + general_summary_prompt_name: str = "v2_general.summary_answer", + workflow_llm_enabled: bool = True, + ) -> None: + self._router = router or V2IntentRouter() + self._policy_resolver = policy_resolver + self._rag_adapter = rag_adapter + self._evidence_assembler = evidence_assembler + self._evidence_gate = evidence_gate or DocsEvidenceGate() + self._docs_summary_prompt_name = docs_summary_prompt_name + self._general_summary_prompt_name = general_summary_prompt_name + self._workflow_llm_enabled = workflow_llm_enabled + self._summary_graph = DocsExplainSummaryGraph(llm) + self._find_files_graph = DocsExplainFindFilesGraph() + self._general_summary_graph = GeneralSummaryGraph(llm) + + async def run(self, context) -> ProcessResult: + route = self._router.route(context.request.message) + rag_session_id = context.session.active_rag_session_id + context.trace.module("process.v2").log( + "intent_routed", + { + "routing_domain": route.routing_domain, + "intent": route.intent, + "subintent": route.subintent, + "normalized_query": route.normalized_query, + "target_terms": route.target_terms, + "anchors": route_anchor_summary(route), + "confidence": route.confidence, + "routing_mode": route.routing_mode, + "llm_router_used": route.llm_router_used, + "reason_short": route.reason_short, + "rag_session_id": rag_session_id, + }, + ) + self._log_step( + context, + "router_resolved", + { + "domain": route.routing_domain, + "intent": route.intent, + "subintent": route.subintent, + "confidence": route.confidence, + }, + ) + self._log_step( + context, + "anchors_extracted", + { + "signal_types": route_anchor_summary(route)["signal_types"], + "endpoint_paths": route.anchors.endpoint_paths, + "target_doc_hints": route.anchors.target_doc_hints, + "matched_aliases": route.anchors.matched_aliases, + "target_terms": route.target_terms, + }, + ) + self._log_step( + context, + "alias_resolution", + { + "resolved_aliases": route.anchors.matched_aliases, + "target_doc_hints": route.anchors.target_doc_hints, + }, + ) + if not rag_session_id: + if route.intent == V2Intent.GENERAL_QA: + answer = "Не могу собрать grounded summary без активной RAG-сессии с проиндексированной документацией." + self._log_step(context, "evidence_gate_checked", {"passed": False, "reason": "missing_rag_session"}) + self._log_step(context, "answer_generated", {"answer_mode": "insufficient_evidence"}) + return ProcessResult(answer=answer) + return ProcessResult(answer="Для процесса v2 нужна активная RAG-сессия проекта с проиндексированной документацией.") + plan = self._policy_resolver.resolve(route) + context.trace.module("process.v2.retrieval_policy").log( + "retrieval_plan_resolved", + {"profile": plan.profile, "layers": plan.layers, "limit": plan.limit, "filters": plan.filters}, + ) + self._log_step( + context, + "retrieval_profile_selected", + {"profile": plan.profile, "layers": plan.layers, "filters": plan.filters}, + ) + seeded_rows = await self._seed_candidates_from_target_hints(rag_session_id, plan.layers, route) + semantic_rows = await self._rag_adapter.fetch_rows(rag_session_id, route.normalized_query, plan) + metadata_rows = self._metadata_lookup_candidates([*seeded_rows, *semantic_rows], route) + rows = self._merge_candidate_rows(seeded_rows, metadata_rows, semantic_rows) + rows = await self._ensure_target_hints_in_pool(rag_session_id, rows, route) + rows = seed_candidates_from_target_hints(rows, route.anchors.target_doc_hints, RagRowIndex(rows)) + self._print_missing_target_hints(route, rows) + context.trace.module("process.v2.rag_retrieval").log( + "rag_rows_fetched", + { + "profile": plan.profile, + "row_count": len(rows), + "rows": [self._trace_row(row) for row in rows], + }, + ) + self._log_step( + context, + "candidate_generation", + { + "query": route.user_query, + "profile": plan.profile, + "details": { + "target_doc_hints": list(route.anchors.target_doc_hints), + "candidates_before_ranking": [row_path(row) for row in rows if row_path(row)], + }, + "resolved_aliases": route.anchors.matched_aliases, + "target_doc_hints": route.anchors.target_doc_hints, + "candidate_docs_before_ranking": [self._trace_row(row) for row in rows[:8]], + "sources": { + "seeded": [self._trace_row(row) for row in seeded_rows[:5]], + "metadata_lookup": [self._trace_row(row) for row in metadata_rows[:5]], + "semantic": [self._trace_row(row) for row in semantic_rows[:5]], + }, + }, + ) + self._log_step( + context, + "retrieval_executed", + { + "query": route.user_query, + "profile": plan.profile, + "row_count": len(rows), + "target_doc_hints": route.anchors.target_doc_hints, + "top_results": [self._trace_row(row) for row in rows[:5]], + }, + ) + if route.subintent == V2Subintent.FIND_FILES: + files = self._evidence_assembler.assemble_files(rows, route) + gate = self._evidence_gate.check_files(route, files) + context.trace.module("process.v2.evidence").log( + "evidence_assembled", + {"mode": "find_files", "file_count": len(files), "files": [file.path for file in files]}, + ) + self._log_step( + context, + "evidence_assembled", + {"mode": "find_files", "primary_file": files[0].path if files else None, "file_count": len(files)}, + ) + self._log_ranking(context, files) + self._log_step( + context, + "evidence_gate_checked", + {"passed": gate.passed, "reason": gate.reason, "answer_mode": gate.answer_mode}, + ) + flow_context = DocsExplainFindFilesContext( + runtime=context, + route=route, + rag_session_id=rag_session_id, + files=files, + gate_decision=gate, + ) + flow_context = await self._find_files_graph.run(flow_context) + self._log_step(context, "answer_generated", {"answer_mode": gate.answer_mode, "answer_length": len(flow_context.answer)}) + return ProcessResult(answer=flow_context.answer) + documents = self._evidence_assembler.assemble_summaries(rows, route) + gate = self._evidence_gate.check_summaries(route, documents) + context.trace.module("process.v2.evidence").log( + "evidence_assembled", + {"mode": "summary", "document_count": len(documents), "documents": [item.path for item in documents]}, + ) + self._log_step( + context, + "evidence_assembled", + {"mode": "summary", "primary_doc": documents[0].path if documents else None, "document_count": len(documents)}, + ) + self._log_ranking(context, documents) + self._log_step( + context, + "evidence_gate_checked", + {"passed": gate.passed, "reason": gate.reason, "answer_mode": gate.answer_mode}, + ) + if route.intent == V2Intent.GENERAL_QA: + flow_context = GeneralSummaryContext( + runtime=context, + route=route, + prompt_name=self._general_summary_prompt_name, + workflow_llm_enabled=self._workflow_llm_enabled, + documents=documents, + gate_decision=gate, + ) + flow_context = await self._general_summary_graph.run(flow_context) + self._log_step(context, "answer_generated", {"answer_mode": gate.answer_mode, "answer_length": len(flow_context.answer)}) + return ProcessResult(answer=flow_context.answer) + flow_context = DocsExplainSummaryContext( + runtime=context, + route=route, + rag_session_id=rag_session_id, + prompt_name=self._docs_summary_prompt_name, + workflow_llm_enabled=self._workflow_llm_enabled, + documents=documents, + gate_decision=gate, + ) + flow_context = await self._summary_graph.run(flow_context) + self._log_step(context, "answer_generated", {"answer_mode": gate.answer_mode, "answer_length": len(flow_context.answer)}) + return ProcessResult(answer=flow_context.answer) + + def _trace_row(self, row: dict) -> dict[str, object]: + metadata = row.get("metadata") or {} + content = str(row.get("content") or "").strip() + return { + "layer": str(row.get("layer") or ""), + "path": str(row.get("path") or ""), + "title": str(row.get("title") or ""), + "document_id": str(metadata.get("document_id") or metadata.get("doc_id") or ""), + "entity_name": str(metadata.get("entity_name") or ""), + "summary_text": str(metadata.get("summary_text") or "")[:400], + "section_path": str(metadata.get("section_path") or ""), + "content_preview": content[:400], + } + + def _log_step(self, context, step: str, payload: dict[str, object]) -> None: + context.trace.module("process.v2.pipeline").log(step, payload) + + def _print_missing_target_hints(self, route, rows: list[dict]) -> None: + if not route.anchors.target_doc_hints: + return + candidate_paths = normalized_path_set(rows) + for hint in route.anchors.target_doc_hints: + if not str(hint or "").strip(): + continue + normalized = normalize_doc_path(hint) + if normalized not in candidate_paths: + print("ERROR: target doc missing from candidates:", normalized) + + async def _ensure_target_hints_in_pool(self, rag_session_id: str, rows: list[dict], route) -> list[dict]: + hints_raw = [str(item).strip() for item in route.anchors.target_doc_hints if str(item or "").strip()] + if not hints_raw: + return rows + pool = normalized_path_set(rows) + missing_hints = [h for h in hints_raw if normalize_doc_path(h) not in pool] + if not missing_hints: + return rows + variant_paths: list[str] = [] + for h in missing_hints: + variant_paths.extend(path_variants_for_rag_query(h)) + variant_paths = list(dict.fromkeys(variant_paths)) + extra_exact = await self._rag_adapter.fetch_exact_paths(rag_session_id, paths=variant_paths, layers=None) + pool2 = normalized_path_set(extra_exact) + still_missing = [h for h in missing_hints if normalize_doc_path(h) not in pool2] + fallback_rows: list[dict] = [] + if still_missing: + needles = [normalize_doc_path(h).split("/")[-1] for h in still_missing] + needles = list(dict.fromkeys(n for n in needles if n)) + if needles: + fallback_rows = await self._rag_adapter.fetch_chunks_by_path_substrings( + rag_session_id, + path_needles=needles, + layers=None, + ) + return merge_row_lists(rows, extra_exact, fallback_rows) + + async def _seed_candidates_from_target_hints(self, rag_session_id: str, layers: list[str], route) -> list[dict]: + del layers # seed по пути должен видеть все слои (иначе D0-only чанки теряются при file_lookup). + hints_raw = [str(item).strip() for item in route.anchors.target_doc_hints if str(item or "").strip()] + if not hints_raw: + return [] + variant_paths: list[str] = [] + for h in hints_raw: + variant_paths.extend(path_variants_for_rag_query(h)) + variant_paths = list(dict.fromkeys(variant_paths)) + exact_rows = await self._rag_adapter.fetch_exact_paths(rag_session_id, paths=variant_paths, layers=None) + paths_found = normalized_path_set(exact_rows) + missing = [h for h in hints_raw if normalize_doc_path(h) not in paths_found] + if not missing: + return exact_rows + needles = [normalize_doc_path(h).split("/")[-1] for h in missing] + needles = list(dict.fromkeys(n for n in needles if n)) + if not needles: + return exact_rows + fallback_rows = await self._rag_adapter.fetch_chunks_by_path_substrings( + rag_session_id, + path_needles=needles, + layers=None, + ) + return merge_row_lists(exact_rows, fallback_rows) + + def _metadata_lookup_candidates(self, rows: list[dict], route) -> list[dict]: + return DocsMetadataLookupIndex(rows).lookup(route) + + def _merge_candidate_rows(self, *groups: list[dict]) -> list[dict]: + return merge_row_lists(*groups) + + def _log_ranking(self, context, items: list) -> None: + top_docs: list[dict[str, object]] = [] + for item in items[:4]: + top_docs.append( + { + "doc": getattr(item, "path", ""), + "score": getattr(item, "score", 0), + "match_reason": getattr(item, "match_reason", ""), + } + ) + context.trace.module("process.v2.pipeline").log( + "ranking_explained", + { + "doc": getattr(item, "path", ""), + "score_breakdown": getattr(item, "score_breakdown", {}), + "score": getattr(item, "score", 0), + "match_reason": getattr(item, "match_reason", ""), + }, + ) + context.trace.module("process.v2.pipeline").log( + "ranking_explained", + { + "top_docs_after_ranking": top_docs, + "ranking_score_breakdown": [ + { + "doc": getattr(item, "path", ""), + "score_breakdown": getattr(item, "score_breakdown", {}), + } + for item in items[:4] + ], + }, + ) diff --git a/src/app/core/agent/processes/v2/prompts.yml b/src/app/core/agent/processes/v2/prompts.yml new file mode 100644 index 0000000..eee92f7 --- /dev/null +++ b/src/app/core/agent/processes/v2/prompts.yml @@ -0,0 +1,8 @@ +namespace: v2_docs_explain + +prompts: + summary_answer: | + Ты объясняешь документацию только на основе найденных SUMMARY-блоков. + Используй только факты из входного контекста. + Если информации мало, прямо скажи об этом и не додумывай детали. + В конце перечисли файлы, на которые ты опирался. diff --git a/src/app/core/agent/processes/v2/retrieval/__init__.py b/src/app/core/agent/processes/v2/retrieval/__init__.py new file mode 100644 index 0000000..11efcb3 --- /dev/null +++ b/src/app/core/agent/processes/v2/retrieval/__init__.py @@ -0,0 +1,17 @@ +from app.core.agent.processes.v2.retrieval.metadata_lookup import DocsMetadataLookupIndex +from app.core.agent.processes.v2.retrieval.policy_resolver import V2RetrievalPolicyResolver +from app.core.agent.processes.v2.retrieval.target_doc_seeding import ( + RagRowIndex, + normalize_doc_path, + seed_candidates_from_target_hints, +) +from app.core.agent.processes.v2.retrieval.v2_rag_adapter import V2RagRetrievalAdapter + +__all__ = [ + "V2RetrievalPolicyResolver", + "V2RagRetrievalAdapter", + "DocsMetadataLookupIndex", + "normalize_doc_path", + "RagRowIndex", + "seed_candidates_from_target_hints", +] diff --git a/src/app/core/agent/processes/v2/retrieval/metadata_lookup.py b/src/app/core/agent/processes/v2/retrieval/metadata_lookup.py new file mode 100644 index 0000000..6125329 --- /dev/null +++ b/src/app/core/agent/processes/v2/retrieval/metadata_lookup.py @@ -0,0 +1,66 @@ +from __future__ import annotations + +import re +from collections import defaultdict + +from app.core.agent.processes.v2.models import V2RouteResult + + +class DocsMetadataLookupIndex: + def __init__(self, rows: list[dict]) -> None: + self._rows_by_path: dict[str, dict] = {} + self._rows_by_basename: dict[str, list[dict]] = defaultdict(list) + self._rows_by_slug: dict[str, list[dict]] = defaultdict(list) + self._rows_by_title_token: dict[str, list[dict]] = defaultdict(list) + self._rows_by_compact: dict[str, list[dict]] = defaultdict(list) + for row in rows: + path = str(row.get("path") or "").strip() + if not path or path in self._rows_by_path: + continue + self._rows_by_path[path] = row + basename = path.split("/")[-1].lower() + slug = basename.removesuffix(".md").removesuffix(".yaml").removesuffix(".yml") + self._rows_by_basename[basename].append(row) + self._rows_by_slug[slug].append(row) + self._rows_by_compact[self._compact(slug)].append(row) + title = str(row.get("title") or "").lower() + for token in self._tokens(title): + self._rows_by_title_token[token].append(row) + self._rows_by_compact[self._compact(title)].append(row) + entity_name = str(dict(row.get("metadata") or {}).get("entity_name") or "").lower() + if entity_name: + self._rows_by_compact[self._compact(entity_name)].append(row) + + def lookup(self, route: V2RouteResult) -> list[dict]: + candidates: list[dict] = [] + seen: set[str] = set() + for path in route.anchors.target_doc_hints: + self._append(candidates, seen, self._rows_by_path.get(path)) + lookup_tokens = list(route.target_terms) + list(route.anchors.matched_aliases) + list(route.anchors.endpoint_paths) + for token in self._tokens(" ".join(lookup_tokens)): + for bucket in ( + self._rows_by_basename.get(token, []), + self._rows_by_slug.get(token, []), + self._rows_by_title_token.get(token, []), + ): + for row in bucket: + self._append(candidates, seen, row) + for compact in {self._compact(item) for item in lookup_tokens if item}: + for row in self._rows_by_compact.get(compact, []): + self._append(candidates, seen, row) + return candidates + + def _append(self, items: list[dict], seen: set[str], row: dict | None) -> None: + if row is None: + return + path = str(row.get("path") or "").strip() + if not path or path in seen: + return + seen.add(path) + items.append(row) + + def _tokens(self, value: str) -> list[str]: + return [token for token in re.split(r"[^a-zA-Zа-яА-Я0-9]+", str(value or "").lower()) if len(token) >= 3] + + def _compact(self, value: str) -> str: + return "".join(self._tokens(value)) diff --git a/src/app/core/agent/processes/v2/retrieval/policy_resolver.py b/src/app/core/agent/processes/v2/retrieval/policy_resolver.py new file mode 100644 index 0000000..bd424d3 --- /dev/null +++ b/src/app/core/agent/processes/v2/retrieval/policy_resolver.py @@ -0,0 +1,118 @@ +"""Intent-aware retrieval policy resolver для процесса v2.""" + +from __future__ import annotations + +from app.core.agent.processes.v2.anchor_signals import anchor_signal_types +from app.core.agent.processes.v2.models import V2AnchorType, V2Intent, V2RouteResult, V2Subintent +from app.core.rag.contracts.enums import RagLayer +from app.core.rag.retrieval.session_retriever import RetrievalPlan + + +class V2RetrievalPolicyResolver: + _SUMMARY_LAYERS = [ + RagLayer.DOCS_DOCUMENT_CATALOG, + RagLayer.DOCS_ENTITY_CATALOG, + RagLayer.DOCS_DOC_CHUNKS, + ] + _GENERAL_LAYERS = [ + RagLayer.DOCS_DOCUMENT_CATALOG, + RagLayer.DOCS_DOC_CHUNKS, + ] + + def resolve(self, route: V2RouteResult) -> RetrievalPlan: + if route.intent == V2Intent.GENERAL_QA: + return RetrievalPlan( + profile="general_qa_grounded_summary", + layers=list(self._GENERAL_LAYERS), + limit=8, + filters=self._general_filters(route), + ) + if route.subintent == V2Subintent.FIND_FILES: + return RetrievalPlan( + profile="file_lookup", + layers=[RagLayer.DOCS_DOCUMENT_CATALOG, RagLayer.DOCS_ENTITY_CATALOG], + limit=12, + filters=self._find_files_filters(route), + ) + return RetrievalPlan( + profile=self._summary_profile(route), + layers=list(self._SUMMARY_LAYERS), + limit=8, + filters=self._summary_filters(route), + ) + + def _summary_profile(self, route: V2RouteResult) -> str: + signals = anchor_signal_types(route) + if len(signals - {V2AnchorType.FIND_FILES}) != 1: + return "docs_summary_generic" + mapping = { + V2AnchorType.API_ENDPOINT: "docs_summary_api_endpoint", + V2AnchorType.ARCHITECTURE: "docs_summary_architecture", + V2AnchorType.LOGIC_FLOW: "docs_summary_logic_flow", + V2AnchorType.DOMAIN_ENTITY: "docs_summary_domain_entity", + } + signal = next(iter(signals - {V2AnchorType.FIND_FILES}), None) + return mapping.get(signal, "docs_summary_generic") + + def _general_filters(self, route: V2RouteResult) -> dict[str, object]: + return { + "prefer_path_prefixes": ["docs/architecture/", "docs/"], + "prefer_like_patterns": ["%README.md%", "%overview%"], + "target_doc_hints": list(route.anchors.target_doc_hints), + } + + def _summary_filters(self, route: V2RouteResult) -> dict[str, object]: + filters: dict[str, object] = { + "prefer_path_prefixes": self._summary_prefixes(route), + "prefer_like_patterns": self._prefer_like_patterns(route), + "target_doc_hints": list(route.anchors.target_doc_hints), + } + if V2AnchorType.API_ENDPOINT in anchor_signal_types(route): + filters["path_prefixes"] = ["docs/api/", "docs/architecture/", "docs/"] + return filters + + def _find_files_filters(self, route: V2RouteResult) -> dict[str, object]: + filters: dict[str, object] = { + "prefer_path_prefixes": self._find_files_prefixes(route), + "prefer_like_patterns": self._prefer_like_patterns(route), + "target_doc_hints": list(route.anchors.target_doc_hints), + } + if route.anchors.target_doc_hints: + filters["prefer_like_patterns"] = [f"%{path.split('/')[-1]}%" for path in route.anchors.target_doc_hints] + return filters + + def _prefer_like_patterns(self, route: V2RouteResult) -> list[str]: + patterns: list[str] = [] + for path in route.anchors.target_doc_hints: + patterns.append(f"%{path.split('/')[-1]}%") + for endpoint in route.anchors.endpoint_paths: + patterns.append(f"%{endpoint}%") + return patterns + + def _find_files_prefixes(self, route: V2RouteResult) -> list[str]: + if route.anchors.target_doc_hints: + prefixes = ["/".join(path.split("/")[:-1]) + "/" for path in route.anchors.target_doc_hints] + return [prefix for prefix in prefixes if prefix] + signals = anchor_signal_types(route) + if V2AnchorType.API_ENDPOINT in signals: + return ["docs/api/", "docs/"] + if V2AnchorType.ARCHITECTURE in signals: + return ["docs/architecture/", "docs/"] + if V2AnchorType.LOGIC_FLOW in signals: + return ["docs/logic/", "docs/"] + if V2AnchorType.DOMAIN_ENTITY in signals: + return ["docs/domains/", "docs/"] + return ["docs/"] + + def _summary_prefixes(self, route: V2RouteResult) -> list[str]: + signals = anchor_signal_types(route) + prefixes: list[str] = [] + if V2AnchorType.API_ENDPOINT in signals: + prefixes.extend(["docs/api/", "docs/"]) + if V2AnchorType.ARCHITECTURE in signals: + prefixes.extend(["docs/architecture/", "docs/"]) + if V2AnchorType.LOGIC_FLOW in signals: + prefixes.extend(["docs/logic/", "docs/architecture/", "docs/"]) + if V2AnchorType.DOMAIN_ENTITY in signals: + prefixes.extend(["docs/domains/", "docs/api/", "docs/architecture/"]) + return list(dict.fromkeys(prefixes or ["docs/"])) diff --git a/src/app/core/agent/processes/v2/retrieval/target_doc_seeding.py b/src/app/core/agent/processes/v2/retrieval/target_doc_seeding.py new file mode 100644 index 0000000..3c8151f --- /dev/null +++ b/src/app/core/agent/processes/v2/retrieval/target_doc_seeding.py @@ -0,0 +1,114 @@ +from __future__ import annotations + + +def normalize_doc_path(path: str | None) -> str: + value = str(path or "").strip().replace("\\", "/") + if not value: + return "" + while "//" in value: + value = value.replace("//", "/") + while value.startswith("./"): + value = value[2:] + value = value.lstrip("/") + docs_idx = value.lower().find("docs/") + if docs_idx >= 0: + value = value[docs_idx:] + elif "/" not in value and value.lower().endswith(".md"): + value = f"docs/{value}" + return value.strip() + + +def row_path(row: dict) -> str: + metadata = dict(row.get("metadata") or {}) + raw = row.get("path") or metadata.get("source_path") or "" + return normalize_doc_path(str(raw)) + + +def normalized_path_set(rows: list[dict]) -> set[str]: + return {path for row in rows if (path := row_path(row))} + + +def path_variants_for_rag_query(path: str | None) -> list[str]: + normalized = normalize_doc_path(path) + if not normalized: + return [] + variants = [normalized] + if normalized.startswith("docs/"): + variants.append(normalized.removeprefix("docs/")) + else: + variants.append(f"docs/{normalized}") + basename = normalized.split("/")[-1] + if basename and basename not in variants: + variants.append(basename) + return list(dict.fromkeys(variants)) + + +def merge_row_lists(*groups: list[dict]) -> list[dict]: + merged: list[dict] = [] + seen: set[tuple[str, str, str]] = set() + for rows in groups: + for row in rows: + metadata = dict(row.get("metadata") or {}) + key = ( + row_path(row), + str(row.get("layer") or ""), + str(metadata.get("section_path") or ""), + ) + if key in seen: + continue + seen.add(key) + merged.append(row) + return merged + + +class RagRowIndex: + def __init__(self, rows: list[dict]) -> None: + self._by_path: dict[str, list[dict]] = {} + self._by_name: dict[str, list[dict]] = {} + for row in rows: + normalized = row_path(row) + if not normalized: + continue + self._by_path.setdefault(normalized.lower(), []).append(row) + basename = normalized.split("/")[-1].lower() + self._by_name.setdefault(basename, []).append(row) + + def lookup(self, hint: str | None) -> list[dict]: + matches: list[dict] = [] + seen_ids: set[int] = set() + for variant in path_variants_for_rag_query(hint): + key = variant.lower() + for row in self._by_path.get(key, []): + row_id = id(row) + if row_id in seen_ids: + continue + seen_ids.add(row_id) + matches.append(row) + basename = normalize_doc_path(hint).split("/")[-1].lower() + for row in self._by_name.get(basename, []): + row_id = id(row) + if row_id in seen_ids: + continue + seen_ids.add(row_id) + matches.append(row) + return matches + + +def seed_candidates_from_target_hints(rows: list[dict], hints: list[str], index: RagRowIndex | None = None) -> list[dict]: + hints_raw = [str(hint).strip() for hint in hints if str(hint or "").strip()] + if not hints_raw or not rows: + return rows + rag_index = index or RagRowIndex(rows) + seeded = [match for hint in hints_raw for match in rag_index.lookup(hint)] + return merge_row_lists(seeded, rows) + + +__all__ = [ + "RagRowIndex", + "merge_row_lists", + "normalize_doc_path", + "normalized_path_set", + "path_variants_for_rag_query", + "row_path", + "seed_candidates_from_target_hints", +] diff --git a/src/app/core/agent/processes/v2/retrieval/v2_rag_adapter.py b/src/app/core/agent/processes/v2/retrieval/v2_rag_adapter.py new file mode 100644 index 0000000..b5ca6d0 --- /dev/null +++ b/src/app/core/agent/processes/v2/retrieval/v2_rag_adapter.py @@ -0,0 +1,33 @@ +"""Адаптер v2 к :class:`RagSessionRetriever` для подстановки в тестах.""" + +from __future__ import annotations + +from app.core.rag.retrieval.session_retriever import RagSessionRetriever, RetrievalPlan + + +class V2RagRetrievalAdapter: + """Обёртка над :class:`RagSessionRetriever` для подмены в тестах.""" + + def __init__(self, retriever: RagSessionRetriever) -> None: + self._retriever = retriever + + async def fetch_rows(self, rag_session_id: str, query_text: str, plan: RetrievalPlan) -> list[dict]: + return await self._retriever.retrieve(rag_session_id, query_text, plan) + + async def fetch_exact_paths(self, rag_session_id: str, *, paths: list[str], layers: list[str] | None = None) -> list[dict]: + return await self._retriever.retrieve_exact_files(rag_session_id, paths=paths, layers=layers) + + async def fetch_chunks_by_path_substrings( + self, + rag_session_id: str, + *, + path_needles: list[str], + layers: list[str] | None = None, + limit: int = 200, + ) -> list[dict]: + return await self._retriever.retrieve_chunks_by_path_substrings( + rag_session_id, + path_needles=path_needles, + layers=layers, + limit=limit, + ) diff --git a/src/app/modules/agent/orchestration/__init__.py b/src/app/core/agent/processes/v2/workflows/__init__.py similarity index 100% rename from src/app/modules/agent/orchestration/__init__.py rename to src/app/core/agent/processes/v2/workflows/__init__.py diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/__init__.py b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/__init__.py new file mode 100644 index 0000000..74d0a49 --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.processes.v2.workflows.docs_explain_find_files.graph import DocsExplainFindFilesGraph + +__all__ = ["DocsExplainFindFilesGraph"] diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/context.py b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/context.py new file mode 100644 index 0000000..51dad0a --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/context.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from dataclasses import dataclass, field + +from app.core.agent.processes.v2.evidence.gate import EvidenceGateDecision +from app.core.agent.processes.v2.models import RetrievedFile, V2RouteResult +from app.core.agent.runtime.execution_context import RuntimeExecutionContext + + +@dataclass(slots=True) +class DocsExplainFindFilesContext: + runtime: RuntimeExecutionContext + route: V2RouteResult + rag_session_id: str + files: list[RetrievedFile] = field(default_factory=list) + gate_decision: EvidenceGateDecision | None = None + answer: str = "" diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/graph.py b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/graph.py new file mode 100644 index 0000000..e9ad91f --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/graph.py @@ -0,0 +1,16 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.workflows.docs_explain_find_files.context import DocsExplainFindFilesContext +from app.core.agent.processes.v2.workflows.docs_explain_find_files.steps.finalize_find_files_answer_step import ( + FinalizeFindFilesAnswerStep, +) +from app.core.agent.processes.v2.workflows.v2_workflow_graph import V2WorkflowGraph + + +class DocsExplainFindFilesGraph(V2WorkflowGraph[DocsExplainFindFilesContext]): + def __init__(self) -> None: + super().__init__( + workflow_id="v2.docs_explain.find_files", + source="workflow.v2.find_files", + steps=[FinalizeFindFilesAnswerStep()], + ) diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/steps/finalize_find_files_answer_step.py b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/steps/finalize_find_files_answer_step.py new file mode 100644 index 0000000..863ae4c --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_find_files/steps/finalize_find_files_answer_step.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.workflows.docs_explain_find_files.context import DocsExplainFindFilesContext +from app.core.agent.utils.workflow import WorkflowStep + + +class FinalizeFindFilesAnswerStep(WorkflowStep[DocsExplainFindFilesContext]): + step_id = "finalize_find_files_answer" + title = "Сборка списка файлов" + + async def run(self, context: DocsExplainFindFilesContext) -> DocsExplainFindFilesContext: + if not context.files: + context.answer = "Не нашёл файлов документации, которые уверенно соответствуют запросу." + return context + if context.gate_decision is not None and context.gate_decision.reason == "low_confidence_shortlist": + context.answer = "\n".join(item.path for item in context.files[:4]) + return context + if len(context.files) == 1: + context.answer = context.files[0].path + return context + context.answer = "\n".join(item.path for item in context.files[:4]) + return context + + def trace_output(self, context: DocsExplainFindFilesContext) -> dict[str, object]: + return {"answer_length": len(context.answer)} diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_summary/__init__.py b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/__init__.py new file mode 100644 index 0000000..99020fa --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.processes.v2.workflows.docs_explain_summary.graph import DocsExplainSummaryGraph + +__all__ = ["DocsExplainSummaryGraph"] diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_summary/context.py b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/context.py new file mode 100644 index 0000000..660dd63 --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/context.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +from dataclasses import dataclass, field + +from app.core.agent.processes.v2.evidence.gate import EvidenceGateDecision +from app.core.agent.processes.v2.models import RetrievedSummary, V2RouteResult +from app.core.agent.runtime.execution_context import RuntimeExecutionContext + + +@dataclass(slots=True) +class DocsExplainSummaryContext: + runtime: RuntimeExecutionContext + route: V2RouteResult + rag_session_id: str + prompt_name: str + workflow_llm_enabled: bool = True + documents: list[RetrievedSummary] = field(default_factory=list) + gate_decision: EvidenceGateDecision | None = None + prompt_input: str = "" + answer: str = "" diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_summary/graph.py b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/graph.py new file mode 100644 index 0000000..d1df68a --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/graph.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.workflows.docs_explain_summary.context import DocsExplainSummaryContext +from app.core.agent.processes.v2.workflows.docs_explain_summary.steps.generate_summary_answer_step import ( + GenerateSummaryAnswerStep, +) +from app.core.agent.processes.v2.workflows.v2_workflow_graph import V2WorkflowGraph +from app.core.agent.utils.llm import AgentLlmService + + +class DocsExplainSummaryGraph(V2WorkflowGraph[DocsExplainSummaryContext]): + def __init__(self, llm: AgentLlmService) -> None: + super().__init__( + workflow_id="v2.docs_explain.summary", + source="workflow.v2.summary", + steps=[GenerateSummaryAnswerStep(llm)], + ) diff --git a/src/app/core/agent/processes/v2/workflows/docs_explain_summary/steps/generate_summary_answer_step.py b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/steps/generate_summary_answer_step.py new file mode 100644 index 0000000..030fa50 --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/docs_explain_summary/steps/generate_summary_answer_step.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +import asyncio +import json + +from app.core.agent.processes.v2.anchor_signals import route_anchor_summary +from app.core.agent.utils.llm import AgentLlmService +from app.core.agent.processes.v2.workflows.docs_explain_summary.context import DocsExplainSummaryContext +from app.core.agent.utils.workflow import WorkflowStep + + +class GenerateSummaryAnswerStep(WorkflowStep[DocsExplainSummaryContext]): + step_id = "generate_summary_answer" + title = "Сборка ответа по summary" + + def __init__(self, llm: AgentLlmService) -> None: + self._llm = llm + + async def run(self, context: DocsExplainSummaryContext) -> DocsExplainSummaryContext: + if context.gate_decision is not None and not context.gate_decision.passed: + context.answer = context.gate_decision.message + return context + if not context.workflow_llm_enabled: + context.answer = self._build_deterministic_answer(context) + return context + if not context.documents: + context.answer = "Не нашёл подходящих SUMMARY-блоков в документации по этому запросу." + return context + context.prompt_input = self._build_prompt_input(context) + request_id = context.runtime.request.request_id + context.answer = await asyncio.to_thread( + self._llm.generate, + context.prompt_name, + context.prompt_input, + log_context=f"agent:{request_id}", + trace=context.runtime.trace.module("workflow.v2.summary.llm"), + ) + return context + + def _build_prompt_input(self, context: DocsExplainSummaryContext) -> str: + blocks = [ + f"Запрос пользователя:\n{context.route.user_query}", + "Сигналы запроса:\n" + json.dumps(route_anchor_summary(context.route), ensure_ascii=False, indent=2), + "Найденные SUMMARY-блоки:", + ] + for index, item in enumerate(context.documents, start=1): + blocks.append( + f"{index}. path: {item.path}\n" + f"title: {item.title}\n" + f"match_reason: {item.match_reason}\n" + f"summary: {item.summary}" + ) + return "\n\n".join(blocks) + + def _build_deterministic_answer(self, context: DocsExplainSummaryContext) -> str: + if not context.documents: + return "Не нашёл подходящих SUMMARY-блоков в документации по этому запросу." + lines = [] + primary = context.documents[0] + lines.append(primary.summary) + lines.append("") + lines.append("Файлы-источники:") + for item in context.documents: + lines.append(f"- {item.path}") + return "\n".join(lines) + + def trace_output(self, context: DocsExplainSummaryContext) -> dict[str, object]: + return {"answer_length": len(context.answer)} diff --git a/src/app/core/agent/processes/v2/workflows/general_summary/__init__.py b/src/app/core/agent/processes/v2/workflows/general_summary/__init__.py new file mode 100644 index 0000000..732779c --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/general_summary/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.processes.v2.workflows.general_summary.graph import GeneralSummaryGraph + +__all__ = ["GeneralSummaryGraph"] diff --git a/src/app/core/agent/processes/v2/workflows/general_summary/context.py b/src/app/core/agent/processes/v2/workflows/general_summary/context.py new file mode 100644 index 0000000..d2b2507 --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/general_summary/context.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +from dataclasses import dataclass, field + +from app.core.agent.processes.v2.evidence.gate import EvidenceGateDecision +from app.core.agent.processes.v2.models import RetrievedSummary, V2RouteResult +from app.core.agent.runtime.execution_context import RuntimeExecutionContext + + +@dataclass(slots=True) +class GeneralSummaryContext: + runtime: RuntimeExecutionContext + route: V2RouteResult + prompt_name: str + workflow_llm_enabled: bool = True + documents: list[RetrievedSummary] = field(default_factory=list) + gate_decision: EvidenceGateDecision | None = None + prompt_input: str = "" + answer: str = "" diff --git a/src/app/core/agent/processes/v2/workflows/general_summary/graph.py b/src/app/core/agent/processes/v2/workflows/general_summary/graph.py new file mode 100644 index 0000000..5441cad --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/general_summary/graph.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.workflows.general_summary.context import GeneralSummaryContext +from app.core.agent.processes.v2.workflows.general_summary.steps.generate_general_summary_answer_step import ( + GenerateGeneralSummaryAnswerStep, +) +from app.core.agent.processes.v2.workflows.v2_workflow_graph import V2WorkflowGraph +from app.core.agent.utils.llm import AgentLlmService + + +class GeneralSummaryGraph(V2WorkflowGraph[GeneralSummaryContext]): + def __init__(self, llm: AgentLlmService) -> None: + super().__init__( + workflow_id="v2.general_qa.summary", + source="workflow.v2.general_summary", + steps=[GenerateGeneralSummaryAnswerStep(llm)], + ) diff --git a/src/app/core/agent/processes/v2/workflows/general_summary/steps/generate_general_summary_answer_step.py b/src/app/core/agent/processes/v2/workflows/general_summary/steps/generate_general_summary_answer_step.py new file mode 100644 index 0000000..5e6945f --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/general_summary/steps/generate_general_summary_answer_step.py @@ -0,0 +1,57 @@ +from __future__ import annotations + +import asyncio + +from app.core.agent.processes.v2.workflows.general_summary.context import GeneralSummaryContext +from app.core.agent.utils.llm import AgentLlmService +from app.core.agent.utils.workflow import WorkflowStep + + +class GenerateGeneralSummaryAnswerStep(WorkflowStep[GeneralSummaryContext]): + step_id = "generate_general_summary_answer" + title = "Общий ответ через LLM" + + def __init__(self, llm: AgentLlmService) -> None: + self._llm = llm + + async def run(self, context: GeneralSummaryContext) -> GeneralSummaryContext: + if context.gate_decision is not None and not context.gate_decision.passed: + context.answer = context.gate_decision.message + return context + if not context.workflow_llm_enabled: + context.answer = self._build_deterministic_answer(context) + return context + context.prompt_input = self._build_prompt_input(context) + request_id = context.runtime.request.request_id + context.answer = await asyncio.to_thread( + self._llm.generate, + context.prompt_name, + context.prompt_input, + log_context=f"agent:{request_id}", + trace=context.runtime.trace.module("workflow.v2.general_summary.llm"), + ) + return context + + def _build_prompt_input(self, context: GeneralSummaryContext) -> str: + blocks = [ + f"Запрос пользователя:\n{context.route.user_query}", + "Опорные документы:", + ] + for index, item in enumerate(context.documents, start=1): + blocks.append( + f"{index}. path: {item.path}\n" + f"title: {item.title}\n" + f"summary: {item.summary}" + ) + return "\n\n".join(blocks) + + def _build_deterministic_answer(self, context: GeneralSummaryContext) -> str: + if not context.documents: + return "В найденной документации нет достаточной опоры для общего summary по запросу." + return "\n".join(item.summary for item in context.documents[:2] if item.summary) + + def trace_input(self, context: GeneralSummaryContext) -> dict[str, object]: + return {"query": context.route.normalized_query} + + def trace_output(self, context: GeneralSummaryContext) -> dict[str, object]: + return {"answer_length": len(context.answer)} diff --git a/src/app/core/agent/processes/v2/workflows/v2_workflow_graph.py b/src/app/core/agent/processes/v2/workflows/v2_workflow_graph.py new file mode 100644 index 0000000..b3db545 --- /dev/null +++ b/src/app/core/agent/processes/v2/workflows/v2_workflow_graph.py @@ -0,0 +1,39 @@ +"""Workflow-граф v2: буфер шаговых логов и один сброс в trace в конце прогона.""" + +from __future__ import annotations + +from typing import Generic, Sequence, TypeVar + +from app.core.agent.utils.workflow.context import WorkflowContext +from app.core.agent.utils.workflow.graph import WorkflowGraph +from app.core.agent.utils.workflow.step import WorkflowStep + + +TContext = TypeVar("TContext", bound=WorkflowContext) + + +class V2WorkflowGraph(WorkflowGraph[TContext]): + """Не логирует step_started/step_completed по отдельности; сбрасывает буфер в ``workflow_trace_flushed``.""" + + async def run(self, context: TContext) -> TContext: + trace = context.runtime.trace.module(self._source) + trace.log("workflow_started", {"workflow_id": self._workflow_id}) + steps_buffer: list[dict[str, object]] = [] + for step in self._steps: + inp = step.trace_input(context) + request_id = context.runtime.request.request_id + await context.runtime.publisher.publish_status( + request_id, + self._source, + f"Шаг workflow: {step.title}.", + {"workflow_id": self._workflow_id, "step_id": step.step_id}, + ) + context = await step.run(context) + out = step.trace_output(context) + steps_buffer.append({"step_id": step.step_id, "title": step.title, "input": inp, "output": out}) + trace.log( + "workflow_trace_flushed", + {"workflow_id": self._workflow_id, "steps": steps_buffer}, + ) + trace.log("workflow_completed", {"workflow_id": self._workflow_id}) + return context diff --git a/src/app/core/agent/runtime/__init__.py b/src/app/core/agent/runtime/__init__.py new file mode 100644 index 0000000..0327fa6 --- /dev/null +++ b/src/app/core/agent/runtime/__init__.py @@ -0,0 +1,13 @@ +from app.core.agent.runtime.agent_runtime import AgentRuntime +from app.core.agent.runtime.execution_context import RuntimeExecutionContext +from app.core.agent.runtime.process_registry import ProcessRegistry +from app.core.agent.runtime.process_runner import ProcessRunner +from app.core.agent.runtime.publisher import RuntimeEventPublisher + +__all__ = [ + "AgentRuntime", + "ProcessRegistry", + "ProcessRunner", + "RuntimeEventPublisher", + "RuntimeExecutionContext", +] diff --git a/src/app/core/agent/runtime/agent_runtime.py b/src/app/core/agent/runtime/agent_runtime.py new file mode 100644 index 0000000..817513e --- /dev/null +++ b/src/app/core/agent/runtime/agent_runtime.py @@ -0,0 +1,107 @@ +from __future__ import annotations + +from datetime import datetime, timezone + +from app.core.api.application.session_service import SessionService +from app.core.api.domain.models.agent_request import AgentRequest +from app.core.api.domain.models.agent_session import AgentSession +from app.core.api.infrastructure.stores.in_memory_request_store import InMemoryRequestStore +from app.core.agent.runtime.execution_context import RuntimeExecutionContext +from app.core.agent.runtime.process_registry import ProcessRegistry +from app.core.agent.runtime.process_runner import ProcessRunner +from app.core.agent.runtime.publisher import RuntimeEventPublisher +from app.infra.exceptions import AppError +from app.infra.observability.module_trace import RequestTraceContext +from app.infra.observability.request_trace_logger import RequestTraceLogger +from app.schemas.common import ErrorPayload, ModuleName +from app.schemas.orchestration import RequestExecutionStatus + + +class AgentRuntime: + def __init__( + self, + request_store: InMemoryRequestStore, + sessions: SessionService, + process_registry: ProcessRegistry, + process_runner: ProcessRunner, + publisher: RuntimeEventPublisher, + trace_logger: RequestTraceLogger, + ) -> None: + self._request_store = request_store + self._sessions = sessions + self._process_registry = process_registry + self._process_runner = process_runner + self._publisher = publisher + self._trace_logger = trace_logger + + async def run(self, request: AgentRequest, session: AgentSession) -> None: + try: + process = self._resolve_process(request.process_version) + self._start_request(request, session) + context = RuntimeExecutionContext( + request=request, + session=session, + publisher=self._publisher, + trace=RequestTraceContext(request_id=request.request_id, logger=self._trace_logger), + ) + await self._announce_start(request.request_id, process.version) + result = await self._process_runner.run(context, process) + request.answer = result.answer + await self._publish_result(request) + self._complete_request(request, session) + except Exception as exc: + await self._fail_request(request, exc) + + def _resolve_process(self, version: str): + process = self._process_registry.get(version) + if process is None: + raise AppError("process_not_found", f"Unsupported process version: {version}", ModuleName.AGENT) + return process + + def _start_request(self, request: AgentRequest, session: AgentSession) -> None: + request.status = RequestExecutionStatus.RUNNING + self._request_store.save(request) + self._trace_logger.start_request(request, session) + + async def _announce_start(self, request_id: str, process_version: str) -> None: + await self._publisher.publish_status(request_id, "runtime", "Запрос принят и поставлен в обработку.") + await self._publisher.publish_status( + request_id, + "runtime", + f"Запускаю процесс {process_version}.", + {"process_version": process_version}, + ) + + async def _publish_result(self, request: AgentRequest) -> None: + await self._publisher.publish_user(request.request_id, "agent", request.answer or "") + await self._publisher.publish_status(request.request_id, "runtime", "Обработка запроса завершена.") + + def _complete_request(self, request: AgentRequest, session: AgentSession) -> None: + session.append_turn(user_message=request.message, assistant_message=request.answer or "") + self._sessions.save(session) + request.status = RequestExecutionStatus.DONE + request.completed_at = datetime.now(timezone.utc) + self._request_store.save(request) + self._trace_logger.complete_request(request) + + async def _fail_request(self, request: AgentRequest, exc: Exception) -> None: + request.status = RequestExecutionStatus.ERROR + request.completed_at = datetime.now(timezone.utc) + request.error = self._build_error_payload(exc) + self._request_store.save(request) + self._trace_logger.fail_request(request) + await self._publisher.publish_status( + request.request_id, + "runtime", + "Во время обработки запроса произошла ошибка.", + {"code": request.error.code}, + ) + + def _build_error_payload(self, exc: Exception) -> ErrorPayload: + if isinstance(exc, AppError): + return ErrorPayload(code=exc.code, desc=exc.desc, module=exc.module) + return ErrorPayload( + code="api_runtime_error", + desc="Agent request failed unexpectedly.", + module=ModuleName.AGENT, + ) diff --git a/src/app/core/agent/runtime/execution_context.py b/src/app/core/agent/runtime/execution_context.py new file mode 100644 index 0000000..b7e1481 --- /dev/null +++ b/src/app/core/agent/runtime/execution_context.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any + +from app.core.api.domain.models.agent_request import AgentRequest +from app.core.api.domain.models.agent_session import AgentSession +from app.infra.observability.module_trace import RequestTraceContext + +if TYPE_CHECKING: + from app.core.agent.runtime.publisher import RuntimeEventPublisher + + +@dataclass(slots=True) +class RuntimeExecutionContext: + request: AgentRequest + session: AgentSession + publisher: "RuntimeEventPublisher" + trace: RequestTraceContext + state: dict[str, Any] = field(default_factory=dict) diff --git a/src/app/core/agent/runtime/process_registry.py b/src/app/core/agent/runtime/process_registry.py new file mode 100644 index 0000000..0591112 --- /dev/null +++ b/src/app/core/agent/runtime/process_registry.py @@ -0,0 +1,13 @@ +from __future__ import annotations + +from collections.abc import Iterable + +from app.core.agent.processes.base import AgentProcess + + +class ProcessRegistry: + def __init__(self, processes: Iterable[AgentProcess]) -> None: + self._items = {process.version: process for process in processes} + + def get(self, version: str) -> AgentProcess | None: + return self._items.get(version) diff --git a/src/app/core/agent/runtime/process_runner.py b/src/app/core/agent/runtime/process_runner.py new file mode 100644 index 0000000..68dd096 --- /dev/null +++ b/src/app/core/agent/runtime/process_runner.py @@ -0,0 +1,9 @@ +from __future__ import annotations + +from app.core.agent.processes.base import AgentProcess, ProcessResult +from app.core.agent.runtime.execution_context import RuntimeExecutionContext + + +class ProcessRunner: + async def run(self, context: RuntimeExecutionContext, process: AgentProcess) -> ProcessResult: + return await process.run(context) diff --git a/src/app/core/agent/runtime/publisher.py b/src/app/core/agent/runtime/publisher.py new file mode 100644 index 0000000..c3d86b1 --- /dev/null +++ b/src/app/core/agent/runtime/publisher.py @@ -0,0 +1,36 @@ +from __future__ import annotations + +from app.core.api.domain.events.client_event import ClientEventRecord +from app.core.api.infrastructure.streaming.sse_event_channel import SseEventChannel +from app.infra.observability.request_trace_logger import RequestTraceLogger +from app.schemas.client_events import ClientEventType + + +class RuntimeEventPublisher: + def __init__(self, channel: SseEventChannel, trace_logger: RequestTraceLogger) -> None: + self._channel = channel + self._trace_logger = trace_logger + + async def publish_status(self, request_id: str, source: str, text: str, payload: dict | None = None) -> None: + await self._publish(request_id, ClientEventType.STATUS, source, text, payload) + + async def publish_user(self, request_id: str, source: str, text: str, payload: dict | None = None) -> None: + await self._publish(request_id, ClientEventType.USER, source, text, payload) + + async def _publish( + self, + request_id: str, + event_type: ClientEventType, + source: str, + text: str, + payload: dict | None = None, + ) -> None: + event = ClientEventRecord( + request_id=request_id, + type=event_type, + source=source, + text=text, + payload=payload or {}, + ) + self._trace_logger.log_event(event) + await self._channel.publish(event) diff --git a/src/app/core/agent/utils/__init__.py b/src/app/core/agent/utils/__init__.py new file mode 100644 index 0000000..60caf92 --- /dev/null +++ b/src/app/core/agent/utils/__init__.py @@ -0,0 +1,3 @@ +from app.core.agent.utils.llm import AgentLlmService, PromptLoader + +__all__ = ["AgentLlmService", "PromptLoader"] diff --git a/src/app/core/agent/utils/llm/__init__.py b/src/app/core/agent/utils/llm/__init__.py new file mode 100644 index 0000000..9dc9368 --- /dev/null +++ b/src/app/core/agent/utils/llm/__init__.py @@ -0,0 +1,4 @@ +from app.core.agent.utils.llm.prompt_loader import PromptLoader +from app.core.agent.utils.llm.service import AgentLlmService + +__all__ = ["AgentLlmService", "PromptLoader"] diff --git a/src/app/core/agent/utils/llm/prompt_loader.py b/src/app/core/agent/utils/llm/prompt_loader.py new file mode 100644 index 0000000..c5d8c95 --- /dev/null +++ b/src/app/core/agent/utils/llm/prompt_loader.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import os +from collections.abc import Iterable +from pathlib import Path + +import yaml + + +class PromptLoader: + def __init__(self, prompts_path: Path | Iterable[Path] | None = None) -> None: + self._paths = self._resolve_paths(prompts_path) + self._prompts = self._load_prompts() + + def load(self, name: str) -> str: + return str(self._prompts.get(name, "") or "").strip() + + def _load_prompts(self) -> dict[str, str]: + merged: dict[str, str] = {} + for path in self._paths: + if not path.is_file(): + continue + payload = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + if not isinstance(payload, dict): + continue + namespace = str(payload.get("namespace") or "").strip() + prompts = payload.get("prompts", payload) + if not isinstance(prompts, dict): + continue + for key, value in prompts.items(): + prompt_name = f"{namespace}.{key}" if namespace else str(key) + merged[prompt_name] = str(value or "") + return merged + + def _resolve_paths(self, prompts_path: Path | Iterable[Path] | None) -> tuple[Path, ...]: + if prompts_path is None: + base = Path(__file__).resolve().parent / "prompts.yml" + env_override = os.getenv("AGENT_PROMPTS_DIR", "").strip() + raw_path = Path(env_override) if env_override else base + return (raw_path / "prompts.yml" if raw_path.is_dir() else raw_path,) + if isinstance(prompts_path, Path): + return (prompts_path,) + return tuple(Path(item) for item in prompts_path) diff --git a/src/app/modules/agent/llm/service.py b/src/app/core/agent/utils/llm/service.py similarity index 89% rename from src/app/modules/agent/llm/service.py rename to src/app/core/agent/utils/llm/service.py index 82e83f5..91ed4b2 100644 --- a/src/app/modules/agent/llm/service.py +++ b/src/app/core/agent/utils/llm/service.py @@ -1,8 +1,10 @@ +from __future__ import annotations + import logging -from app.modules.agent.observability.module_trace import ModuleTrace -from app.modules.agent.llm.prompt_loader import PromptLoader -from app.modules.shared.gigachat.client import GigaChatClient +from app.core.agent.utils.llm.prompt_loader import PromptLoader +from app.core.shared.gigachat.client import GigaChatClient +from app.infra.observability.module_trace import ModuleTrace LOGGER = logging.getLogger(__name__) diff --git a/src/app/core/agent/utils/traces/__init__.py b/src/app/core/agent/utils/traces/__init__.py new file mode 100644 index 0000000..dc5b892 --- /dev/null +++ b/src/app/core/agent/utils/traces/__init__.py @@ -0,0 +1 @@ +"""Shared trace helpers will live here.""" diff --git a/src/app/core/agent/utils/workflow/__init__.py b/src/app/core/agent/utils/workflow/__init__.py new file mode 100644 index 0000000..805308d --- /dev/null +++ b/src/app/core/agent/utils/workflow/__init__.py @@ -0,0 +1,9 @@ +from app.core.agent.utils.workflow.context import WorkflowContext +from app.core.agent.utils.workflow.graph import WorkflowGraph +from app.core.agent.utils.workflow.step import WorkflowStep + +__all__ = [ + "WorkflowContext", + "WorkflowGraph", + "WorkflowStep", +] diff --git a/src/app/core/agent/utils/workflow/context.py b/src/app/core/agent/utils/workflow/context.py new file mode 100644 index 0000000..e8d6f42 --- /dev/null +++ b/src/app/core/agent/utils/workflow/context.py @@ -0,0 +1,9 @@ +from __future__ import annotations + +from typing import Protocol + +from app.core.agent.runtime.execution_context import RuntimeExecutionContext + + +class WorkflowContext(Protocol): + runtime: RuntimeExecutionContext diff --git a/src/app/core/agent/utils/workflow/graph.py b/src/app/core/agent/utils/workflow/graph.py new file mode 100644 index 0000000..e0098ea --- /dev/null +++ b/src/app/core/agent/utils/workflow/graph.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +from typing import Generic, Sequence, TypeVar + +from app.core.agent.utils.workflow.context import WorkflowContext +from app.core.agent.utils.workflow.step import WorkflowStep + + +TContext = TypeVar("TContext", bound=WorkflowContext) + + +class WorkflowGraph(Generic[TContext]): + def __init__(self, workflow_id: str, source: str, steps: Sequence[WorkflowStep[TContext]]) -> None: + self._workflow_id = workflow_id + self._source = source + self._steps = tuple(steps) + + async def run(self, context: TContext) -> TContext: + trace = context.runtime.trace.module(self._source) + trace.log("workflow_started", {"workflow_id": self._workflow_id}) + for step in self._steps: + context = await self._run_step(context, step) + trace.log("workflow_completed", {"workflow_id": self._workflow_id}) + return context + + async def _run_step(self, context: TContext, step: WorkflowStep[TContext]) -> TContext: + request_id = context.runtime.request.request_id + trace = context.runtime.trace.module(self._source) + trace.log( + "step_started", + {"workflow_id": self._workflow_id, "step_id": step.step_id, "input": step.trace_input(context)}, + ) + await context.runtime.publisher.publish_status( + request_id, + self._source, + f"Шаг workflow: {step.title}.", + {"workflow_id": self._workflow_id, "step_id": step.step_id}, + ) + context = await step.run(context) + trace.log( + "step_completed", + {"workflow_id": self._workflow_id, "step_id": step.step_id, "output": step.trace_output(context)}, + ) + return context diff --git a/src/app/core/agent/utils/workflow/step.py b/src/app/core/agent/utils/workflow/step.py new file mode 100644 index 0000000..db6fb6d --- /dev/null +++ b/src/app/core/agent/utils/workflow/step.py @@ -0,0 +1,22 @@ +from __future__ import annotations + +from abc import ABC, abstractmethod +from typing import Any, Generic, TypeVar + + +TContext = TypeVar("TContext") + + +class WorkflowStep(ABC, Generic[TContext]): + step_id = "" + title = "" + + @abstractmethod + async def run(self, context: TContext) -> TContext: + raise NotImplementedError + + def trace_input(self, context: TContext) -> dict[str, Any]: + return {} + + def trace_output(self, context: TContext) -> dict[str, Any]: + return {} diff --git a/src/app/modules/api/__init__.py b/src/app/core/api/__init__.py similarity index 100% rename from src/app/modules/api/__init__.py rename to src/app/core/api/__init__.py diff --git a/src/app/modules/api/application/request_service.py b/src/app/core/api/application/request_service.py similarity index 61% rename from src/app/modules/api/application/request_service.py rename to src/app/core/api/application/request_service.py index ca4da41..8df37c7 100644 --- a/src/app/modules/api/application/request_service.py +++ b/src/app/core/api/application/request_service.py @@ -2,11 +2,11 @@ from __future__ import annotations import asyncio -from app.modules.api.domain.models.agent_request import AgentRequest -from app.modules.api.infrastructure.ids.request_id_factory import RequestIdFactory -from app.modules.api.infrastructure.stores.in_memory_request_store import InMemoryRequestStore -from app.modules.api.application.session_service import SessionService -from app.modules.agent.orchestration.facade import OrchestrationFacade +from app.core.api.domain.models.agent_request import AgentRequest +from app.core.api.infrastructure.ids.request_id_factory import RequestIdFactory +from app.core.api.infrastructure.stores.in_memory_request_store import InMemoryRequestStore +from app.core.api.application.session_service import SessionService +from app.core.agent.runtime import AgentRuntime class RequestService: @@ -15,12 +15,12 @@ class RequestService: request_store: InMemoryRequestStore, request_ids: RequestIdFactory, sessions: SessionService, - orchestration: OrchestrationFacade, + runtime: AgentRuntime, ) -> None: self._request_store = request_store self._request_ids = request_ids self._sessions = sessions - self._orchestration = orchestration + self._runtime = runtime async def create(self, session_id: str, message: str, process_version: str) -> AgentRequest: session = self._sessions.get(session_id) @@ -31,7 +31,7 @@ class RequestService: process_version=process_version, ) self._request_store.save(request) - asyncio.create_task(self._orchestration.run(request, session)) + asyncio.create_task(self._runtime.run(request, session)) return request def get(self, request_id: str) -> AgentRequest | None: diff --git a/src/app/core/api/application/session_bootstrap_service.py b/src/app/core/api/application/session_bootstrap_service.py new file mode 100644 index 0000000..e55a1b1 --- /dev/null +++ b/src/app/core/api/application/session_bootstrap_service.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from dataclasses import dataclass + +from app.core.api.application.session_service import SessionService +from app.core.api.domain.models.agent_session import AgentSession +from app.core.rag.indexing import IndexJob +from app.core.rag.module import RagModule + + +@dataclass(slots=True) +class BootstrappedAgentSession: + session: AgentSession + index_job: IndexJob + + +class SessionBootstrapService: + def __init__(self, sessions: SessionService, rag: RagModule) -> None: + self._sessions = sessions + self._rag = rag + + async def create(self, project_id: str, files: list[dict]) -> BootstrappedAgentSession: + rag_session, index_job = await self._rag.create_session(project_id=project_id, files=files) + session = self._sessions.create(rag_session_id=rag_session.rag_session_id) + return BootstrappedAgentSession(session=session, index_job=index_job) diff --git a/src/app/core/api/application/session_service.py b/src/app/core/api/application/session_service.py new file mode 100644 index 0000000..0991aa5 --- /dev/null +++ b/src/app/core/api/application/session_service.py @@ -0,0 +1,26 @@ +from __future__ import annotations + +from app.infra.exceptions import AppError +from app.core.api.domain.models.agent_session import AgentSession +from app.core.api.infrastructure.ids.session_id_factory import SessionIdFactory +from app.core.api.infrastructure.stores.in_memory_session_store import InMemorySessionStore +from app.schemas.common import ModuleName + + +class SessionService: + def __init__(self, store: InMemorySessionStore, ids: SessionIdFactory) -> None: + self._store = store + self._ids = ids + + def create(self, rag_session_id: str | None = None) -> AgentSession: + session = AgentSession.create(self._ids.create(), rag_session_id=rag_session_id) + return self._store.save(session) + + def get(self, session_id: str) -> AgentSession: + session = self._store.get(session_id) + if session is None: + raise AppError("session_not_found", f"Agent session not found: {session_id}", ModuleName.BACKEND) + return session + + def save(self, session: AgentSession) -> AgentSession: + return self._store.save(session) diff --git a/src/app/modules/api/application/stream_service.py b/src/app/core/api/application/stream_service.py similarity index 80% rename from src/app/modules/api/application/stream_service.py rename to src/app/core/api/application/stream_service.py index 1986f5c..7a38b1f 100644 --- a/src/app/modules/api/application/stream_service.py +++ b/src/app/core/api/application/stream_service.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.core.exceptions import AppError -from app.modules.api.infrastructure.streaming.sse_encoder import SseEncoder -from app.modules.api.infrastructure.streaming.sse_event_channel import SseEventChannel +from app.infra.exceptions import AppError +from app.core.api.infrastructure.streaming.sse_encoder import SseEncoder +from app.core.api.infrastructure.streaming.sse_event_channel import SseEventChannel from app.schemas.common import ModuleName diff --git a/src/app/core/api/controllers/rag_public_controller.py b/src/app/core/api/controllers/rag_public_controller.py new file mode 100644 index 0000000..c1b9881 --- /dev/null +++ b/src/app/core/api/controllers/rag_public_controller.py @@ -0,0 +1,33 @@ +from __future__ import annotations + +from app.core.api.infrastructure.streaming.sse_response_builder import build_sse_response +from app.core.rag.module import RagModule +from app.core.shared.messaging import EventBus +from app.schemas.rag_sessions import RagSessionJobResponse + + +class RagPublicController: + def __init__(self, rag: RagModule) -> None: + self._rag = rag + + def get_job(self, rag_session_id: str, index_job_id: str) -> RagSessionJobResponse: + job = self._rag.get_session_job(rag_session_id, index_job_id) + return RagSessionJobResponse( + rag_session_id=rag_session_id, + index_job_id=job.index_job_id, + status=job.status, + indexed_files=job.indexed_files, + failed_files=job.failed_files, + cache_hit_files=job.cache_hit_files, + cache_miss_files=job.cache_miss_files, + error=job.error.model_dump(mode="json") if job.error else None, + ) + + async def stream_job_events(self, rag_session_id: str, index_job_id: str): + channel_id, queue = await self._rag.subscribe_session_job_events(rag_session_id, index_job_id) + return build_sse_response( + queue, + encoder=EventBus.as_sse, + unsubscribe=lambda: self._rag.unsubscribe_job_events(channel_id, queue), + stop_on_event="terminal", + ) diff --git a/src/app/modules/api/controllers/request_controller.py b/src/app/core/api/controllers/request_controller.py similarity index 92% rename from src/app/modules/api/controllers/request_controller.py rename to src/app/core/api/controllers/request_controller.py index c2f1841..233806f 100644 --- a/src/app/modules/api/controllers/request_controller.py +++ b/src/app/core/api/controllers/request_controller.py @@ -1,7 +1,7 @@ from __future__ import annotations -from app.core.exceptions import AppError -from app.modules.api.application.request_service import RequestService +from app.infra.exceptions import AppError +from app.core.api.application.request_service import RequestService from app.schemas.agent_api import AgentRequestCreateRequest, AgentRequestQueuedResponse, AgentRequestStateResponse from app.schemas.common import ModuleName diff --git a/src/app/core/api/controllers/session_controller.py b/src/app/core/api/controllers/session_controller.py new file mode 100644 index 0000000..c7b1915 --- /dev/null +++ b/src/app/core/api/controllers/session_controller.py @@ -0,0 +1,23 @@ +from __future__ import annotations + +from app.core.api.application.session_bootstrap_service import SessionBootstrapService +from app.schemas.agent_api import CreateAgentSessionRequest, CreateAgentSessionResponse + + +class SessionController: + def __init__(self, service: SessionBootstrapService) -> None: + self._service = service + + async def create_session(self, request: CreateAgentSessionRequest) -> CreateAgentSessionResponse: + result = await self._service.create( + project_id=request.project_id, + files=[item.model_dump() for item in request.files], + ) + session = result.session + return CreateAgentSessionResponse( + session_id=session.session_id, + rag_session_id=session.active_rag_session_id or "", + index_job_id=result.index_job.index_job_id, + status=result.index_job.status, + created_at=session.created_at, + ) diff --git a/src/app/core/api/controllers/stream_controller.py b/src/app/core/api/controllers/stream_controller.py new file mode 100644 index 0000000..355fda1 --- /dev/null +++ b/src/app/core/api/controllers/stream_controller.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from app.core.api.infrastructure.streaming.sse_response_builder import build_sse_response +from app.core.api.application.stream_service import StreamService + + +class StreamController: + def __init__(self, service: StreamService) -> None: + self._service = service + + async def stream(self, request_id: str): + queue = await self._service.subscribe(request_id) + return build_sse_response( + queue, + encoder=self._service.encode, + unsubscribe=lambda: self._service.unsubscribe(request_id, queue), + ) diff --git a/src/app/modules/api/domain/events/client_event.py b/src/app/core/api/domain/events/client_event.py similarity index 100% rename from src/app/modules/api/domain/events/client_event.py rename to src/app/core/api/domain/events/client_event.py diff --git a/src/app/modules/api/domain/models/agent_request.py b/src/app/core/api/domain/models/agent_request.py similarity index 100% rename from src/app/modules/api/domain/models/agent_request.py rename to src/app/core/api/domain/models/agent_request.py diff --git a/src/app/core/api/domain/models/agent_session.py b/src/app/core/api/domain/models/agent_session.py new file mode 100644 index 0000000..32c2e1f --- /dev/null +++ b/src/app/core/api/domain/models/agent_session.py @@ -0,0 +1,35 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import datetime, timezone + +from app.core.api.domain.models.agent_session_message import AgentSessionMessage + + +@dataclass(slots=True) +class AgentSession: + session_id: str + active_rag_session_id: str | None + created_at: datetime + updated_at: datetime + messages: list[AgentSessionMessage] = field(default_factory=list) + + @classmethod + def create(cls, session_id: str, rag_session_id: str | None = None) -> "AgentSession": + now = datetime.now(timezone.utc) + return cls( + session_id=session_id, + active_rag_session_id=rag_session_id, + created_at=now, + updated_at=now, + ) + + def append_turn(self, user_message: str, assistant_message: str, route_result=None) -> None: + self._append_message("user", user_message) + self._append_message("assistant", assistant_message) + self.updated_at = datetime.now(timezone.utc) + + def _append_message(self, role: str, text: str) -> None: + value = text.strip() + if value: + self.messages.append(AgentSessionMessage.create(role, value)) diff --git a/src/app/core/api/domain/models/agent_session_message.py b/src/app/core/api/domain/models/agent_session_message.py new file mode 100644 index 0000000..88fd2de --- /dev/null +++ b/src/app/core/api/domain/models/agent_session_message.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import Literal + + +SessionMessageRole = Literal["user", "assistant"] + + +@dataclass(slots=True) +class AgentSessionMessage: + role: SessionMessageRole + text: str + created_at: datetime + + @classmethod + def create(cls, role: SessionMessageRole, text: str) -> "AgentSessionMessage": + return cls(role=role, text=text, created_at=datetime.now(timezone.utc)) diff --git a/src/app/modules/api/infrastructure/ids/request_id_factory.py b/src/app/core/api/infrastructure/ids/request_id_factory.py similarity index 100% rename from src/app/modules/api/infrastructure/ids/request_id_factory.py rename to src/app/core/api/infrastructure/ids/request_id_factory.py diff --git a/src/app/modules/api/infrastructure/ids/session_id_factory.py b/src/app/core/api/infrastructure/ids/session_id_factory.py similarity index 100% rename from src/app/modules/api/infrastructure/ids/session_id_factory.py rename to src/app/core/api/infrastructure/ids/session_id_factory.py diff --git a/src/app/modules/api/infrastructure/stores/in_memory_request_store.py b/src/app/core/api/infrastructure/stores/in_memory_request_store.py similarity index 87% rename from src/app/modules/api/infrastructure/stores/in_memory_request_store.py rename to src/app/core/api/infrastructure/stores/in_memory_request_store.py index 5551330..843236c 100644 --- a/src/app/modules/api/infrastructure/stores/in_memory_request_store.py +++ b/src/app/core/api/infrastructure/stores/in_memory_request_store.py @@ -2,7 +2,7 @@ from __future__ import annotations from threading import Lock -from app.modules.api.domain.models.agent_request import AgentRequest +from app.core.api.domain.models.agent_request import AgentRequest class InMemoryRequestStore: diff --git a/src/app/modules/api/infrastructure/stores/in_memory_session_store.py b/src/app/core/api/infrastructure/stores/in_memory_session_store.py similarity index 87% rename from src/app/modules/api/infrastructure/stores/in_memory_session_store.py rename to src/app/core/api/infrastructure/stores/in_memory_session_store.py index 1e13564..fe80dc7 100644 --- a/src/app/modules/api/infrastructure/stores/in_memory_session_store.py +++ b/src/app/core/api/infrastructure/stores/in_memory_session_store.py @@ -2,7 +2,7 @@ from __future__ import annotations from threading import Lock -from app.modules.api.domain.models.agent_session import AgentSession +from app.core.api.domain.models.agent_session import AgentSession class InMemorySessionStore: diff --git a/src/app/modules/api/infrastructure/streaming/replay_buffer.py b/src/app/core/api/infrastructure/streaming/replay_buffer.py similarity index 89% rename from src/app/modules/api/infrastructure/streaming/replay_buffer.py rename to src/app/core/api/infrastructure/streaming/replay_buffer.py index dad9125..a90935a 100644 --- a/src/app/modules/api/infrastructure/streaming/replay_buffer.py +++ b/src/app/core/api/infrastructure/streaming/replay_buffer.py @@ -2,7 +2,7 @@ from __future__ import annotations from collections import defaultdict -from app.modules.api.domain.events.client_event import ClientEventRecord +from app.core.api.domain.events.client_event import ClientEventRecord class ReplayBuffer: diff --git a/src/app/modules/api/infrastructure/streaming/sse_encoder.py b/src/app/core/api/infrastructure/streaming/sse_encoder.py similarity index 87% rename from src/app/modules/api/infrastructure/streaming/sse_encoder.py rename to src/app/core/api/infrastructure/streaming/sse_encoder.py index 0817394..de616af 100644 --- a/src/app/modules/api/infrastructure/streaming/sse_encoder.py +++ b/src/app/core/api/infrastructure/streaming/sse_encoder.py @@ -2,7 +2,7 @@ from __future__ import annotations import json -from app.modules.api.domain.events.client_event import ClientEventRecord +from app.core.api.domain.events.client_event import ClientEventRecord class SseEncoder: diff --git a/src/app/modules/api/infrastructure/streaming/sse_event_channel.py b/src/app/core/api/infrastructure/streaming/sse_event_channel.py similarity index 90% rename from src/app/modules/api/infrastructure/streaming/sse_event_channel.py rename to src/app/core/api/infrastructure/streaming/sse_event_channel.py index dddfd56..2bdd20c 100644 --- a/src/app/modules/api/infrastructure/streaming/sse_event_channel.py +++ b/src/app/core/api/infrastructure/streaming/sse_event_channel.py @@ -3,8 +3,8 @@ from __future__ import annotations import asyncio from collections import defaultdict -from app.modules.api.domain.events.client_event import ClientEventRecord -from app.modules.api.infrastructure.streaming.replay_buffer import ReplayBuffer +from app.core.api.domain.events.client_event import ClientEventRecord +from app.core.api.infrastructure.streaming.replay_buffer import ReplayBuffer class SseEventChannel: diff --git a/src/app/core/api/infrastructure/streaming/sse_response_builder.py b/src/app/core/api/infrastructure/streaming/sse_response_builder.py new file mode 100644 index 0000000..8545172 --- /dev/null +++ b/src/app/core/api/infrastructure/streaming/sse_response_builder.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +from collections.abc import Awaitable, Callable + +from fastapi.responses import StreamingResponse + + +def build_sse_response( + queue, + *, + encoder: Callable[[object], str], + unsubscribe: Callable[[], Awaitable[None]], + heartbeat_seconds: int = 10, + stop_on_event: str | None = None, +) -> StreamingResponse: + async def event_stream(): + import asyncio + + try: + while True: + try: + event = await asyncio.wait_for(queue.get(), timeout=heartbeat_seconds) + yield encoder(event) + if stop_on_event and getattr(event, "name", None) == stop_on_event: + break + except asyncio.TimeoutError: + yield ": keepalive\n\n" + finally: + await unsubscribe() + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache, no-transform", + "Connection": "keep-alive", + "X-Accel-Buffering": "no", + }, + ) diff --git a/src/app/core/api/module.py b/src/app/core/api/module.py new file mode 100644 index 0000000..efff8a1 --- /dev/null +++ b/src/app/core/api/module.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +from fastapi import APIRouter + +from app.core.api.application.session_bootstrap_service import SessionBootstrapService +from app.core.api.application.request_service import RequestService +from app.core.api.application.stream_service import StreamService +from app.core.api.controllers.request_controller import RequestController +from app.core.api.controllers.rag_public_controller import RagPublicController +from app.core.api.controllers.session_controller import SessionController +from app.core.api.controllers.stream_controller import StreamController +from app.core.api.public_router import build_agent_public_router +from app.core.api.rag_public_router import build_rag_public_router +from app.core.rag.module import RagModule + + +class ApiModule: + def __init__( + self, + session_bootstrap: SessionBootstrapService, + requests: RequestService, + streams: StreamService, + rag: RagModule, + ) -> None: + self._sessions = SessionController(session_bootstrap) + self._requests = RequestController(requests) + self._streams = StreamController(streams) + self._rag_public = RagPublicController(rag) + + def public_router(self) -> APIRouter: + router = APIRouter() + router.include_router( + build_agent_public_router( + sessions=self._sessions, + requests=self._requests, + streams=self._streams, + ) + ) + router.include_router(build_rag_public_router(self._rag_public)) + return router diff --git a/src/app/modules/api/public_router.py b/src/app/core/api/public_router.py similarity index 53% rename from src/app/modules/api/public_router.py rename to src/app/core/api/public_router.py index 0bfd27c..0cc780d 100644 --- a/src/app/modules/api/public_router.py +++ b/src/app/core/api/public_router.py @@ -2,21 +2,19 @@ from __future__ import annotations from fastapi import APIRouter -from app.modules.api.controllers.request_controller import RequestController -from app.modules.api.controllers.session_controller import SessionController -from app.modules.api.controllers.stream_controller import StreamController +from app.core.api.controllers.request_controller import RequestController +from app.core.api.controllers.session_controller import SessionController +from app.core.api.controllers.stream_controller import StreamController from app.schemas.agent_api import ( AgentRequestCreateRequest, AgentRequestQueuedResponse, AgentRequestStateResponse, - BindRagSessionRequest, - BindRagSessionResponse, + CreateAgentSessionRequest, CreateAgentSessionResponse, - ResetAgentSessionResponse, ) -def build_public_router( +def build_agent_public_router( sessions: SessionController, requests: RequestController, streams: StreamController, @@ -24,16 +22,8 @@ def build_public_router( router = APIRouter(tags=["agent-api"]) @router.post("/api/agent/sessions", response_model=CreateAgentSessionResponse) - async def create_session() -> CreateAgentSessionResponse: - return sessions.create_session() - - @router.post("/api/agent/sessions/{session_id}/rag", response_model=BindRagSessionResponse) - async def bind_rag_session(session_id: str, request: BindRagSessionRequest) -> BindRagSessionResponse: - return sessions.bind_rag_session(session_id, request) - - @router.post("/api/agent/sessions/{session_id}/reset", response_model=ResetAgentSessionResponse) - async def reset_session(session_id: str) -> ResetAgentSessionResponse: - return sessions.reset_session(session_id) + async def create_session(request: CreateAgentSessionRequest) -> CreateAgentSessionResponse: + return await sessions.create_session(request) @router.post("/api/agent/requests", response_model=AgentRequestQueuedResponse) async def create_request(request: AgentRequestCreateRequest) -> AgentRequestQueuedResponse: diff --git a/src/app/core/api/rag_public_router.py b/src/app/core/api/rag_public_router.py new file mode 100644 index 0000000..3976221 --- /dev/null +++ b/src/app/core/api/rag_public_router.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +from fastapi import APIRouter + +from app.core.api.controllers.rag_public_controller import RagPublicController +from app.schemas.rag_sessions import RagSessionJobResponse + + +def build_rag_public_router(controller: RagPublicController) -> APIRouter: + router = APIRouter(tags=["rag"]) + + @router.get("/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}", response_model=RagSessionJobResponse) + async def rag_session_job(rag_session_id: str, index_job_id: str) -> RagSessionJobResponse: + return controller.get_job(rag_session_id, index_job_id) + + @router.get("/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}/events") + async def rag_session_job_events(rag_session_id: str, index_job_id: str): + return await controller.stream_job_events(rag_session_id, index_job_id) + + return router diff --git a/src/app/core/application.py b/src/app/core/application.py new file mode 100644 index 0000000..723aaaa --- /dev/null +++ b/src/app/core/application.py @@ -0,0 +1,127 @@ +import logging +from pathlib import Path + +from app.core.agent.processes import V1Process, V2Process +from app.core.agent.processes.v2 import V2IntentRouter +from app.core.agent.processes.v2.evidence.assembler import DocsEvidenceAssembler +from app.core.agent.processes.v2.retrieval.policy_resolver import V2RetrievalPolicyResolver +from app.core.agent.processes.v2.retrieval.v2_rag_adapter import V2RagRetrievalAdapter +from app.core.rag.retrieval.session_retriever import RagSessionRetriever +from app.core.agent.runtime import AgentRuntime, ProcessRegistry, ProcessRunner, RuntimeEventPublisher +from app.core.agent.utils.llm import AgentLlmService, PromptLoader +from app.core.api.module import ApiModule +from app.core.api.application.session_bootstrap_service import SessionBootstrapService +from app.core.api.application.request_service import RequestService +from app.core.api.application.session_service import SessionService +from app.core.api.application.stream_service import StreamService +from app.core.api.infrastructure.ids.request_id_factory import RequestIdFactory +from app.core.api.infrastructure.ids.session_id_factory import SessionIdFactory +from app.core.api.infrastructure.stores.in_memory_request_store import InMemoryRequestStore +from app.core.api.infrastructure.stores.in_memory_session_store import InMemorySessionStore +from app.core.api.infrastructure.streaming.sse_event_channel import SseEventChannel +from app.core.rag.module import RagModule +from app.core.rag.embedding.gigachat_embedder import GigaChatEmbedder +from app.core.rag.persistence import RagRepository +from app.core.shared.database import DatabaseReadiness, bootstrap_database +from app.core.shared.messaging import EventBus +from app.core.shared.story_context import StoryContextSchemaRepository +from app.infra.observability import RequestTraceLogger +from app.core.shared.resilience import RetryExecutor + + +class ModularApplication: + def __init__(self) -> None: + self.database_readiness = DatabaseReadiness() + self.events = EventBus() + self.retry = RetryExecutor() + self.rag_repository = RagRepository() + self.story_context_schema_repository = StoryContextSchemaRepository() + + self.rag = RagModule( + event_bus=self.events, + retry=self.retry, + repository=self.rag_repository, + ensure_ready=self.database_readiness.require_ready, + ) + from app.core.shared.gigachat.client import GigaChatClient + from app.core.shared.gigachat.settings import GigaChatSettings + from app.core.shared.gigachat.token_provider import GigaChatTokenProvider + + _giga_settings = GigaChatSettings.from_env() + _giga_client = GigaChatClient(_giga_settings, GigaChatTokenProvider(_giga_settings)) + _v1_prompt_loader = PromptLoader( + Path(__file__).resolve().parent / "agent/processes/v1/workflow/flow_main/prompts.yml" + ) + _v2_prompt_loader = PromptLoader( + [ + Path(__file__).resolve().parent / "agent/processes/v2/prompts.yml", + Path(__file__).resolve().parent / "agent/processes/v2/general_prompts.yml", + Path(__file__).resolve().parent / "agent/processes/v2/intent_router/routers/prompts.yml", + ] + ) + self._v1_llm = AgentLlmService(client=_giga_client, prompts=_v1_prompt_loader) + self._v2_llm = AgentLlmService(client=_giga_client, prompts=_v2_prompt_loader) + _v2_embedder = GigaChatEmbedder(_giga_client) + _v2_rag_retriever = RagSessionRetriever(repository=self.rag_repository, embedder=_v2_embedder) + _v2_rag_adapter = V2RagRetrievalAdapter(_v2_rag_retriever) + _v2_evidence = DocsEvidenceAssembler() + _v2_policy = V2RetrievalPolicyResolver() + + self.agent_sessions = InMemorySessionStore() + self.agent_requests = InMemoryRequestStore() + self.agent_events = SseEventChannel() + self.agent_trace_logger = RequestTraceLogger(Path("runtime_traces/agent_requests")) + _publisher = RuntimeEventPublisher(self.agent_events, self.agent_trace_logger) + _session_service = SessionService( + store=self.agent_sessions, + ids=SessionIdFactory(), + ) + _session_bootstrap = SessionBootstrapService(_session_service, self.rag) + _process_registry = ProcessRegistry( + [ + V1Process(self._v1_llm), + V2Process( + self._v2_llm, + policy_resolver=_v2_policy, + rag_adapter=_v2_rag_adapter, + evidence_assembler=_v2_evidence, + router=V2IntentRouter(llm=self._v2_llm), + workflow_llm_enabled=True, + ), + ] + ) + _runtime = AgentRuntime( + request_store=self.agent_requests, + sessions=_session_service, + process_registry=_process_registry, + process_runner=ProcessRunner(), + publisher=_publisher, + trace_logger=self.agent_trace_logger, + ) + _request_service = RequestService( + request_store=self.agent_requests, + request_ids=RequestIdFactory(), + sessions=_session_service, + runtime=_runtime, + ) + self.api = ApiModule( + session_bootstrap=_session_bootstrap, + requests=_request_service, + streams=StreamService(self.agent_events, request_exists=lambda request_id: self.agent_requests.get(request_id) is not None), + rag=self.rag, + ) + + def startup(self) -> None: + try: + bootstrap_database( + self.rag_repository, + self.story_context_schema_repository, + ) + except Exception as exc: + logging.exception("Database bootstrap failed. Starting backend in degraded mode.") + self.database_readiness.mark_unavailable(exc) + return + self.database_readiness.mark_ready() + + def health_payload(self) -> dict[str, str]: + return self.database_readiness.health_payload() diff --git a/src/app/modules/contracts.py b/src/app/core/contracts.py similarity index 100% rename from src/app/modules/contracts.py rename to src/app/core/contracts.py diff --git a/src/app/modules/rag/README.md b/src/app/core/rag/README.md similarity index 98% rename from src/app/modules/rag/README.md rename to src/app/core/rag/README.md index 51644fc..1bbdd26 100644 --- a/src/app/modules/rag/README.md +++ b/src/app/core/rag/README.md @@ -348,4 +348,4 @@ sequenceDiagram - В первой итерации реализованы `DOCS D1-D4`. - В первой итерации реализованы `CODE C0-C3`. - `C4-C6` зафиксированы в контракте и зарезервированы под следующие этапы. -- Текущие `rag_session` и `rag_repo` работают как facade/adapter поверх нового пакета `rag`. +- Текущий `rag_session` работает как facade поверх нового пакета `rag`. diff --git a/src/app/modules/rag/__init__.py b/src/app/core/rag/__init__.py similarity index 90% rename from src/app/modules/rag/__init__.py rename to src/app/core/rag/__init__.py index 153b59a..415890c 100644 --- a/src/app/modules/rag/__init__.py +++ b/src/app/core/rag/__init__.py @@ -1,4 +1,4 @@ -from app.modules.rag.contracts import ( +from app.core.rag.contracts import ( DocKind, EvidenceLink, EvidenceType, diff --git a/src/app/core/rag/contracts/__init__.py b/src/app/core/rag/contracts/__init__.py new file mode 100644 index 0000000..fc1f4cf --- /dev/null +++ b/src/app/core/rag/contracts/__init__.py @@ -0,0 +1,17 @@ +from app.core.rag.contracts.documents import RagDocument, RagSource, RagSpan +from app.core.rag.contracts.enums import DocKind, EvidenceType, RagLayer, RetrievalMode +from app.core.rag.contracts.evidence import EvidenceLink +from app.core.rag.contracts.retrieval import RetrievalItem, RetrievalQuery + +__all__ = [ + "DocKind", + "EvidenceLink", + "EvidenceType", + "RagDocument", + "RagLayer", + "RagSource", + "RagSpan", + "RetrievalItem", + "RetrievalMode", + "RetrievalQuery", +] diff --git a/src/app/modules/rag/contracts/documents.py b/src/app/core/rag/contracts/documents.py similarity index 97% rename from src/app/modules/rag/contracts/documents.py rename to src/app/core/rag/contracts/documents.py index e53fa4f..452925f 100644 --- a/src/app/modules/rag/contracts/documents.py +++ b/src/app/core/rag/contracts/documents.py @@ -3,7 +3,7 @@ from __future__ import annotations from dataclasses import asdict, dataclass, field from hashlib import sha256 -from app.modules.rag.contracts.evidence import EvidenceLink +from app.core.rag.contracts.evidence import EvidenceLink @dataclass(slots=True) diff --git a/src/app/modules/rag/contracts/enums.py b/src/app/core/rag/contracts/enums.py similarity index 94% rename from src/app/modules/rag/contracts/enums.py rename to src/app/core/rag/contracts/enums.py index 8f2333c..e298ad0 100644 --- a/src/app/modules/rag/contracts/enums.py +++ b/src/app/core/rag/contracts/enums.py @@ -8,6 +8,7 @@ class RagLayer: DOCS_ENTITY_CATALOG = "D3_ENTITY_CATALOG" DOCS_WORKFLOW_INDEX = "D4_WORKFLOW_INDEX" DOCS_RELATION_GRAPH = "D5_RELATION_GRAPH" + DOCS_INTEGRATION_INDEX = "D6_INTEGRATION_INDEX" CODE_SOURCE_CHUNKS = "C0_SOURCE_CHUNKS" CODE_SYMBOL_CATALOG = "C1_SYMBOL_CATALOG" CODE_DEPENDENCY_GRAPH = "C2_DEPENDENCY_GRAPH" diff --git a/src/app/modules/rag/contracts/evidence.py b/src/app/core/rag/contracts/evidence.py similarity index 100% rename from src/app/modules/rag/contracts/evidence.py rename to src/app/core/rag/contracts/evidence.py diff --git a/src/app/modules/rag/contracts/retrieval.py b/src/app/core/rag/contracts/retrieval.py similarity index 100% rename from src/app/modules/rag/contracts/retrieval.py rename to src/app/core/rag/contracts/retrieval.py diff --git a/src/app/modules/rag/embedding/__init__.py b/src/app/core/rag/embedding/__init__.py similarity index 55% rename from src/app/modules/rag/embedding/__init__.py rename to src/app/core/rag/embedding/__init__.py index e87086d..e9eb5ed 100644 --- a/src/app/modules/rag/embedding/__init__.py +++ b/src/app/core/rag/embedding/__init__.py @@ -1,5 +1,5 @@ """Эмбеддинги для RAG (GigaChat и др.).""" -from app.modules.rag.embedding.gigachat_embedder import GigaChatEmbedder +from app.core.rag.embedding.gigachat_embedder import GigaChatEmbedder __all__ = ["GigaChatEmbedder"] diff --git a/src/app/modules/rag/embedding/gigachat_embedder.py b/src/app/core/rag/embedding/gigachat_embedder.py similarity index 81% rename from src/app/modules/rag/embedding/gigachat_embedder.py rename to src/app/core/rag/embedding/gigachat_embedder.py index 78b7af9..48eec36 100644 --- a/src/app/modules/rag/embedding/gigachat_embedder.py +++ b/src/app/core/rag/embedding/gigachat_embedder.py @@ -1,6 +1,6 @@ """Эмбеддер на базе GigaChat API.""" -from app.modules.shared.gigachat.client import GigaChatClient +from app.core.shared.gigachat.client import GigaChatClient class GigaChatEmbedder: diff --git a/src/app/core/rag/indexing/__init__.py b/src/app/core/rag/indexing/__init__.py new file mode 100644 index 0000000..7e2665d --- /dev/null +++ b/src/app/core/rag/indexing/__init__.py @@ -0,0 +1,21 @@ +__all__ = ["IndexJob", "IndexJobStore", "IndexingOrchestrator", "RagService"] + + +def __getattr__(name: str): + if name == "IndexJob": + from app.core.rag.indexing.job_store import IndexJob + + return IndexJob + if name == "IndexJobStore": + from app.core.rag.indexing.job_store import IndexJobStore + + return IndexJobStore + if name == "IndexingOrchestrator": + from app.core.rag.indexing.orchestrator import IndexingOrchestrator + + return IndexingOrchestrator + if name == "RagService": + from app.core.rag.indexing.service import RagService + + return RagService + raise AttributeError(name) diff --git a/src/app/modules/rag/indexing/code/code_text/chunker.py b/src/app/core/rag/indexing/code/code_text/chunker.py similarity index 100% rename from src/app/modules/rag/indexing/code/code_text/chunker.py rename to src/app/core/rag/indexing/code/code_text/chunker.py diff --git a/src/app/modules/rag/indexing/code/code_text/document_builder.py b/src/app/core/rag/indexing/code/code_text/document_builder.py similarity index 76% rename from src/app/modules/rag/indexing/code/code_text/document_builder.py rename to src/app/core/rag/indexing/code/code_text/document_builder.py index 6d053ee..564b102 100644 --- a/src/app/modules/rag/indexing/code/code_text/document_builder.py +++ b/src/app/core/rag/indexing/code/code_text/document_builder.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.modules.rag.contracts import RagDocument, RagLayer, RagSource, RagSpan -from app.modules.rag.indexing.code.code_text.chunker import CodeChunk -from app.modules.rag.retrieval.test_filter import is_test_path +from app.core.rag.contracts import RagDocument, RagLayer, RagSource, RagSpan +from app.core.rag.indexing.code.code_text.chunker import CodeChunk +from app.core.rag.retrieval.test_filter import is_test_path class CodeTextDocumentBuilder: diff --git a/src/app/modules/rag/indexing/code/edges/dataflow_document_builder.py b/src/app/core/rag/indexing/code/edges/dataflow_document_builder.py similarity index 86% rename from src/app/modules/rag/indexing/code/edges/dataflow_document_builder.py rename to src/app/core/rag/indexing/code/edges/dataflow_document_builder.py index fa6a943..bbfbd97 100644 --- a/src/app/modules/rag/indexing/code/edges/dataflow_document_builder.py +++ b/src/app/core/rag/indexing/code/edges/dataflow_document_builder.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.modules.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan -from app.modules.rag.indexing.code.edges.dataflow_slice_builder import DataflowSlice -from app.modules.rag.retrieval.test_filter import is_test_path +from app.core.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan +from app.core.rag.indexing.code.edges.dataflow_slice_builder import DataflowSlice +from app.core.rag.retrieval.test_filter import is_test_path class DataflowDocumentBuilder: diff --git a/src/app/modules/rag/indexing/code/edges/dataflow_slice_builder.py b/src/app/core/rag/indexing/code/edges/dataflow_slice_builder.py similarity index 98% rename from src/app/modules/rag/indexing/code/edges/dataflow_slice_builder.py rename to src/app/core/rag/indexing/code/edges/dataflow_slice_builder.py index fa8dfa6..7022f6f 100644 --- a/src/app/modules/rag/indexing/code/edges/dataflow_slice_builder.py +++ b/src/app/core/rag/indexing/code/edges/dataflow_slice_builder.py @@ -3,8 +3,8 @@ from __future__ import annotations from dataclasses import dataclass from hashlib import sha256 -from app.modules.rag.indexing.code.edges.extractor import PyEdge -from app.modules.rag.indexing.code.symbols.extractor import PySymbol +from app.core.rag.indexing.code.edges.extractor import PyEdge +from app.core.rag.indexing.code.symbols.extractor import PySymbol _IGNORE_PREFIXES = ("logger", "logging", "print", "stdlib", "typing", "dataclasses") diff --git a/src/app/modules/rag/indexing/code/edges/document_builder.py b/src/app/core/rag/indexing/code/edges/document_builder.py similarity index 82% rename from src/app/modules/rag/indexing/code/edges/document_builder.py rename to src/app/core/rag/indexing/code/edges/document_builder.py index 6bf3bb7..2007e60 100644 --- a/src/app/modules/rag/indexing/code/edges/document_builder.py +++ b/src/app/core/rag/indexing/code/edges/document_builder.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.modules.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan -from app.modules.rag.indexing.code.edges.extractor import PyEdge -from app.modules.rag.retrieval.test_filter import is_test_path +from app.core.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan +from app.core.rag.indexing.code.edges.extractor import PyEdge +from app.core.rag.retrieval.test_filter import is_test_path class EdgeDocumentBuilder: diff --git a/src/app/modules/rag/indexing/code/edges/extractor.py b/src/app/core/rag/indexing/code/edges/extractor.py similarity index 100% rename from src/app/modules/rag/indexing/code/edges/extractor.py rename to src/app/core/rag/indexing/code/edges/extractor.py diff --git a/src/app/modules/rag/indexing/code/entrypoints/document_builder.py b/src/app/core/rag/indexing/code/entrypoints/document_builder.py similarity index 90% rename from src/app/modules/rag/indexing/code/entrypoints/document_builder.py rename to src/app/core/rag/indexing/code/entrypoints/document_builder.py index f5e0f63..2e80c99 100644 --- a/src/app/modules/rag/indexing/code/entrypoints/document_builder.py +++ b/src/app/core/rag/indexing/code/entrypoints/document_builder.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.modules.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan -from app.modules.rag.indexing.code.entrypoints.registry import Entrypoint -from app.modules.rag.retrieval.test_filter import is_test_path +from app.core.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan +from app.core.rag.indexing.code.entrypoints.registry import Entrypoint +from app.core.rag.retrieval.test_filter import is_test_path class EntrypointDocumentBuilder: diff --git a/src/app/modules/rag/indexing/code/entrypoints/execution_trace_builder.py b/src/app/core/rag/indexing/code/entrypoints/execution_trace_builder.py similarity index 95% rename from src/app/modules/rag/indexing/code/entrypoints/execution_trace_builder.py rename to src/app/core/rag/indexing/code/entrypoints/execution_trace_builder.py index 8c40dc1..566e9fa 100644 --- a/src/app/modules/rag/indexing/code/entrypoints/execution_trace_builder.py +++ b/src/app/core/rag/indexing/code/entrypoints/execution_trace_builder.py @@ -3,9 +3,9 @@ from __future__ import annotations from dataclasses import dataclass from hashlib import sha256 -from app.modules.rag.indexing.code.edges.extractor import PyEdge -from app.modules.rag.indexing.code.entrypoints.registry import Entrypoint -from app.modules.rag.indexing.code.symbols.extractor import PySymbol +from app.core.rag.indexing.code.edges.extractor import PyEdge +from app.core.rag.indexing.code.entrypoints.registry import Entrypoint +from app.core.rag.indexing.code.symbols.extractor import PySymbol @dataclass(slots=True) diff --git a/src/app/modules/rag/indexing/code/entrypoints/execution_trace_document_builder.py b/src/app/core/rag/indexing/code/entrypoints/execution_trace_document_builder.py similarity index 85% rename from src/app/modules/rag/indexing/code/entrypoints/execution_trace_document_builder.py rename to src/app/core/rag/indexing/code/entrypoints/execution_trace_document_builder.py index 6075d06..b57b29a 100644 --- a/src/app/modules/rag/indexing/code/entrypoints/execution_trace_document_builder.py +++ b/src/app/core/rag/indexing/code/entrypoints/execution_trace_document_builder.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.modules.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan -from app.modules.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTrace -from app.modules.rag.retrieval.test_filter import is_test_path +from app.core.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan +from app.core.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTrace +from app.core.rag.retrieval.test_filter import is_test_path class ExecutionTraceDocumentBuilder: diff --git a/src/app/modules/rag/indexing/code/entrypoints/fastapi_detector.py b/src/app/core/rag/indexing/code/entrypoints/fastapi_detector.py similarity index 97% rename from src/app/modules/rag/indexing/code/entrypoints/fastapi_detector.py rename to src/app/core/rag/indexing/code/entrypoints/fastapi_detector.py index 75f7b85..bfbebea 100644 --- a/src/app/modules/rag/indexing/code/entrypoints/fastapi_detector.py +++ b/src/app/core/rag/indexing/code/entrypoints/fastapi_detector.py @@ -3,7 +3,7 @@ from __future__ import annotations import ast from hashlib import sha256 -from app.modules.rag.indexing.code.entrypoints.registry import Entrypoint +from app.core.rag.indexing.code.entrypoints.registry import Entrypoint class FastApiEntrypointDetector: diff --git a/src/app/modules/rag/indexing/code/entrypoints/flask_detector.py b/src/app/core/rag/indexing/code/entrypoints/flask_detector.py similarity index 93% rename from src/app/modules/rag/indexing/code/entrypoints/flask_detector.py rename to src/app/core/rag/indexing/code/entrypoints/flask_detector.py index 25c0c27..099d679 100644 --- a/src/app/modules/rag/indexing/code/entrypoints/flask_detector.py +++ b/src/app/core/rag/indexing/code/entrypoints/flask_detector.py @@ -2,7 +2,7 @@ from __future__ import annotations from hashlib import sha256 -from app.modules.rag.indexing.code.entrypoints.registry import Entrypoint +from app.core.rag.indexing.code.entrypoints.registry import Entrypoint class FlaskEntrypointDetector: diff --git a/src/app/modules/rag/indexing/code/entrypoints/registry.py b/src/app/core/rag/indexing/code/entrypoints/registry.py similarity index 100% rename from src/app/modules/rag/indexing/code/entrypoints/registry.py rename to src/app/core/rag/indexing/code/entrypoints/registry.py diff --git a/src/app/modules/rag/indexing/code/entrypoints/typer_click_detector.py b/src/app/core/rag/indexing/code/entrypoints/typer_click_detector.py similarity index 93% rename from src/app/modules/rag/indexing/code/entrypoints/typer_click_detector.py rename to src/app/core/rag/indexing/code/entrypoints/typer_click_detector.py index 139f948..c4ca391 100644 --- a/src/app/modules/rag/indexing/code/entrypoints/typer_click_detector.py +++ b/src/app/core/rag/indexing/code/entrypoints/typer_click_detector.py @@ -2,7 +2,7 @@ from __future__ import annotations from hashlib import sha256 -from app.modules.rag.indexing.code.entrypoints.registry import Entrypoint +from app.core.rag.indexing.code.entrypoints.registry import Entrypoint class TyperClickEntrypointDetector: diff --git a/src/app/modules/rag/indexing/code/file_filter.py b/src/app/core/rag/indexing/code/file_filter.py similarity index 85% rename from src/app/modules/rag/indexing/code/file_filter.py rename to src/app/core/rag/indexing/code/file_filter.py index ed4953f..5a11568 100644 --- a/src/app/modules/rag/indexing/code/file_filter.py +++ b/src/app/core/rag/indexing/code/file_filter.py @@ -2,7 +2,7 @@ from __future__ import annotations from pathlib import PurePosixPath -from app.modules.rag.indexing.common.path_filter import should_skip_indexing_path +from app.core.rag.indexing.common.path_filter import should_skip_indexing_path class PythonFileFilter: diff --git a/src/app/modules/rag/indexing/code/pipeline.py b/src/app/core/rag/indexing/code/pipeline.py similarity index 62% rename from src/app/modules/rag/indexing/code/pipeline.py rename to src/app/core/rag/indexing/code/pipeline.py index 794e326..a5165cf 100644 --- a/src/app/modules/rag/indexing/code/pipeline.py +++ b/src/app/core/rag/indexing/code/pipeline.py @@ -1,25 +1,25 @@ from __future__ import annotations -from app.modules.rag.contracts import RagDocument, RagSource -from app.modules.rag.indexing.code.code_text.chunker import CodeTextChunker -from app.modules.rag.indexing.code.code_text.document_builder import CodeTextDocumentBuilder -from app.modules.rag.indexing.code.edges.dataflow_document_builder import DataflowDocumentBuilder -from app.modules.rag.indexing.code.edges.dataflow_slice_builder import DataflowSliceBuilder -from app.modules.rag.indexing.code.edges.document_builder import EdgeDocumentBuilder -from app.modules.rag.indexing.code.edges.extractor import EdgeExtractor -from app.modules.rag.indexing.code.entrypoints.document_builder import EntrypointDocumentBuilder -from app.modules.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTraceBuilder -from app.modules.rag.indexing.code.entrypoints.execution_trace_document_builder import ExecutionTraceDocumentBuilder -from app.modules.rag.indexing.code.entrypoints.fastapi_detector import FastApiEntrypointDetector -from app.modules.rag.indexing.code.entrypoints.flask_detector import FlaskEntrypointDetector -from app.modules.rag.indexing.code.entrypoints.registry import EntrypointDetectorRegistry -from app.modules.rag.indexing.code.entrypoints.typer_click_detector import TyperClickEntrypointDetector -from app.modules.rag.indexing.code.file_filter import PythonFileFilter -from app.modules.rag.indexing.code.roles.builder import SemanticRoleBuilder -from app.modules.rag.indexing.code.roles.document_builder import SemanticRoleDocumentBuilder -from app.modules.rag.indexing.code.symbols.ast_parser import PythonAstParser -from app.modules.rag.indexing.code.symbols.document_builder import SymbolDocumentBuilder -from app.modules.rag.indexing.code.symbols.extractor import SymbolExtractor +from app.core.rag.contracts import RagDocument, RagSource +from app.core.rag.indexing.code.code_text.chunker import CodeTextChunker +from app.core.rag.indexing.code.code_text.document_builder import CodeTextDocumentBuilder +from app.core.rag.indexing.code.edges.dataflow_document_builder import DataflowDocumentBuilder +from app.core.rag.indexing.code.edges.dataflow_slice_builder import DataflowSliceBuilder +from app.core.rag.indexing.code.edges.document_builder import EdgeDocumentBuilder +from app.core.rag.indexing.code.edges.extractor import EdgeExtractor +from app.core.rag.indexing.code.entrypoints.document_builder import EntrypointDocumentBuilder +from app.core.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTraceBuilder +from app.core.rag.indexing.code.entrypoints.execution_trace_document_builder import ExecutionTraceDocumentBuilder +from app.core.rag.indexing.code.entrypoints.fastapi_detector import FastApiEntrypointDetector +from app.core.rag.indexing.code.entrypoints.flask_detector import FlaskEntrypointDetector +from app.core.rag.indexing.code.entrypoints.registry import EntrypointDetectorRegistry +from app.core.rag.indexing.code.entrypoints.typer_click_detector import TyperClickEntrypointDetector +from app.core.rag.indexing.code.file_filter import PythonFileFilter +from app.core.rag.indexing.code.roles.builder import SemanticRoleBuilder +from app.core.rag.indexing.code.roles.document_builder import SemanticRoleDocumentBuilder +from app.core.rag.indexing.code.symbols.ast_parser import PythonAstParser +from app.core.rag.indexing.code.symbols.document_builder import SymbolDocumentBuilder +from app.core.rag.indexing.code.symbols.extractor import SymbolExtractor class CodeIndexingPipeline: diff --git a/src/app/core/rag/indexing/code/roles/__init__.py b/src/app/core/rag/indexing/code/roles/__init__.py new file mode 100644 index 0000000..773543f --- /dev/null +++ b/src/app/core/rag/indexing/code/roles/__init__.py @@ -0,0 +1,9 @@ +from app.core.rag.indexing.code.roles.builder import SemanticRoleBuilder +from app.core.rag.indexing.code.roles.document_builder import SemanticRoleDocumentBuilder +from app.core.rag.indexing.code.roles.models import SemanticRoleRecord + +__all__ = [ + "SemanticRoleBuilder", + "SemanticRoleDocumentBuilder", + "SemanticRoleRecord", +] diff --git a/src/app/modules/rag/indexing/code/roles/builder.py b/src/app/core/rag/indexing/code/roles/builder.py similarity index 72% rename from src/app/modules/rag/indexing/code/roles/builder.py rename to src/app/core/rag/indexing/code/roles/builder.py index bbb02de..421156d 100644 --- a/src/app/modules/rag/indexing/code/roles/builder.py +++ b/src/app/core/rag/indexing/code/roles/builder.py @@ -1,11 +1,11 @@ from __future__ import annotations -from app.modules.rag.indexing.code.edges.dataflow_slice_builder import DataflowSlice -from app.modules.rag.indexing.code.edges.extractor import PyEdge -from app.modules.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTrace -from app.modules.rag.indexing.code.roles.models import SemanticRoleRecord -from app.modules.rag.indexing.code.roles.scorer import SemanticRoleScorer -from app.modules.rag.indexing.code.symbols.extractor import PySymbol +from app.core.rag.indexing.code.edges.dataflow_slice_builder import DataflowSlice +from app.core.rag.indexing.code.edges.extractor import PyEdge +from app.core.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTrace +from app.core.rag.indexing.code.roles.models import SemanticRoleRecord +from app.core.rag.indexing.code.roles.scorer import SemanticRoleScorer +from app.core.rag.indexing.code.symbols.extractor import PySymbol class SemanticRoleBuilder: diff --git a/src/app/modules/rag/indexing/code/roles/document_builder.py b/src/app/core/rag/indexing/code/roles/document_builder.py similarity index 86% rename from src/app/modules/rag/indexing/code/roles/document_builder.py rename to src/app/core/rag/indexing/code/roles/document_builder.py index 30c97f8..edf883d 100644 --- a/src/app/modules/rag/indexing/code/roles/document_builder.py +++ b/src/app/core/rag/indexing/code/roles/document_builder.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.modules.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan -from app.modules.rag.indexing.code.roles.models import SemanticRoleRecord -from app.modules.rag.retrieval.test_filter import is_test_path +from app.core.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource, RagSpan +from app.core.rag.indexing.code.roles.models import SemanticRoleRecord +from app.core.rag.retrieval.test_filter import is_test_path class SemanticRoleDocumentBuilder: diff --git a/src/app/modules/rag/indexing/code/roles/models.py b/src/app/core/rag/indexing/code/roles/models.py similarity index 100% rename from src/app/modules/rag/indexing/code/roles/models.py rename to src/app/core/rag/indexing/code/roles/models.py diff --git a/src/app/modules/rag/indexing/code/roles/scorer.py b/src/app/core/rag/indexing/code/roles/scorer.py similarity index 95% rename from src/app/modules/rag/indexing/code/roles/scorer.py rename to src/app/core/rag/indexing/code/roles/scorer.py index 9a3dd33..81f2943 100644 --- a/src/app/modules/rag/indexing/code/roles/scorer.py +++ b/src/app/core/rag/indexing/code/roles/scorer.py @@ -2,11 +2,11 @@ from __future__ import annotations from dataclasses import dataclass, field -from app.modules.rag.indexing.code.edges.dataflow_slice_builder import DataflowSlice -from app.modules.rag.indexing.code.edges.extractor import PyEdge -from app.modules.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTrace -from app.modules.rag.indexing.code.roles.models import SemanticRoleRecord -from app.modules.rag.indexing.code.symbols.extractor import PySymbol +from app.core.rag.indexing.code.edges.dataflow_slice_builder import DataflowSlice +from app.core.rag.indexing.code.edges.extractor import PyEdge +from app.core.rag.indexing.code.entrypoints.execution_trace_builder import ExecutionTrace +from app.core.rag.indexing.code.roles.models import SemanticRoleRecord +from app.core.rag.indexing.code.symbols.extractor import PySymbol _ROLE_PRIORITY = [ "handler", diff --git a/src/app/modules/rag/indexing/code/symbols/ast_parser.py b/src/app/core/rag/indexing/code/symbols/ast_parser.py similarity index 100% rename from src/app/modules/rag/indexing/code/symbols/ast_parser.py rename to src/app/core/rag/indexing/code/symbols/ast_parser.py diff --git a/src/app/modules/rag/indexing/code/symbols/document_builder.py b/src/app/core/rag/indexing/code/symbols/document_builder.py similarity index 82% rename from src/app/modules/rag/indexing/code/symbols/document_builder.py rename to src/app/core/rag/indexing/code/symbols/document_builder.py index 98b8ea3..0a71e73 100644 --- a/src/app/modules/rag/indexing/code/symbols/document_builder.py +++ b/src/app/core/rag/indexing/code/symbols/document_builder.py @@ -1,8 +1,8 @@ from __future__ import annotations -from app.modules.rag.contracts import RagDocument, RagLayer, RagSource, RagSpan -from app.modules.rag.indexing.code.symbols.extractor import PySymbol -from app.modules.rag.retrieval.test_filter import is_test_path +from app.core.rag.contracts import RagDocument, RagLayer, RagSource, RagSpan +from app.core.rag.indexing.code.symbols.extractor import PySymbol +from app.core.rag.retrieval.test_filter import is_test_path class SymbolDocumentBuilder: diff --git a/src/app/modules/rag/indexing/code/symbols/extractor.py b/src/app/core/rag/indexing/code/symbols/extractor.py similarity index 100% rename from src/app/modules/rag/indexing/code/symbols/extractor.py rename to src/app/core/rag/indexing/code/symbols/extractor.py diff --git a/src/app/modules/rag/indexing/common/document_upserter.py b/src/app/core/rag/indexing/common/document_upserter.py similarity index 81% rename from src/app/modules/rag/indexing/common/document_upserter.py rename to src/app/core/rag/indexing/common/document_upserter.py index 7bf2dba..80ad891 100644 --- a/src/app/modules/rag/indexing/common/document_upserter.py +++ b/src/app/core/rag/indexing/common/document_upserter.py @@ -1,7 +1,7 @@ from __future__ import annotations -from app.modules.rag.contracts import RagDocument -from app.modules.rag.persistence.repository import RagRepository +from app.core.rag.contracts import RagDocument +from app.core.rag.persistence.repository import RagRepository class RagDocumentUpserter: diff --git a/src/app/modules/rag/indexing/common/path_filter.py b/src/app/core/rag/indexing/common/path_filter.py similarity index 100% rename from src/app/modules/rag/indexing/common/path_filter.py rename to src/app/core/rag/indexing/common/path_filter.py diff --git a/src/app/modules/rag/indexing/common/report.py b/src/app/core/rag/indexing/common/report.py similarity index 100% rename from src/app/modules/rag/indexing/common/report.py rename to src/app/core/rag/indexing/common/report.py diff --git a/src/app/modules/rag/indexing/docs/chunkers/markdown_chunker.py b/src/app/core/rag/indexing/docs/chunkers/markdown_chunker.py similarity index 96% rename from src/app/modules/rag/indexing/docs/chunkers/markdown_chunker.py rename to src/app/core/rag/indexing/docs/chunkers/markdown_chunker.py index 2c218e5..507ace3 100644 --- a/src/app/modules/rag/indexing/docs/chunkers/markdown_chunker.py +++ b/src/app/core/rag/indexing/docs/chunkers/markdown_chunker.py @@ -2,7 +2,7 @@ from __future__ import annotations from dataclasses import dataclass -from app.modules.rag.indexing.docs.chunkers.text_chunker import DocTextChunker +from app.core.rag.indexing.docs.chunkers.text_chunker import DocTextChunker @dataclass(slots=True) diff --git a/src/app/modules/rag/indexing/docs/chunkers/text_chunker.py b/src/app/core/rag/indexing/docs/chunkers/text_chunker.py similarity index 100% rename from src/app/modules/rag/indexing/docs/chunkers/text_chunker.py rename to src/app/core/rag/indexing/docs/chunkers/text_chunker.py diff --git a/src/app/modules/rag/indexing/docs/classifier.py b/src/app/core/rag/indexing/docs/classifier.py similarity index 91% rename from src/app/modules/rag/indexing/docs/classifier.py rename to src/app/core/rag/indexing/docs/classifier.py index b7129e0..56b089c 100644 --- a/src/app/modules/rag/indexing/docs/classifier.py +++ b/src/app/core/rag/indexing/docs/classifier.py @@ -2,7 +2,7 @@ from __future__ import annotations from pathlib import PurePosixPath -from app.modules.rag.contracts import DocKind +from app.core.rag.contracts import DocKind class DocsClassifier: diff --git a/src/app/core/rag/indexing/docs/content_parser.py b/src/app/core/rag/indexing/docs/content_parser.py new file mode 100644 index 0000000..bd0cdfa --- /dev/null +++ b/src/app/core/rag/indexing/docs/content_parser.py @@ -0,0 +1,61 @@ +from __future__ import annotations + +import re + +from app.core.rag.indexing.docs.chunkers.markdown_chunker import SectionChunk +from app.core.rag.indexing.docs.models import ParsedDocument, ParsedSubsection + + +class DocsContentParser: + def parse(self, frontmatter: dict, body: str, *, fallback_title: str, sections: list[SectionChunk]) -> ParsedDocument: + summary = self._summary(sections) + detail_sections = self._detail_sections(sections) + details = "\n\n".join(section.content for section in detail_sections).strip() + title = str(frontmatter.get("title") or "").strip() or self._title_from_body(body) or fallback_title + return ParsedDocument( + frontmatter=frontmatter, + title=title, + summary=summary, + details=details, + body=body.strip(), + detail_sections=detail_sections, + ) + + def _summary(self, sections: list[SectionChunk]) -> str: + for section in sections: + if section.section_title.strip().lower() == "summary": + return section.content.strip() + return "" + + def _detail_sections(self, sections: list[SectionChunk]) -> list[ParsedSubsection]: + result = [ + ParsedSubsection( + title=section.section_title, + content=section.content, + path=section.section_path, + ) + for section in sections + if self._is_details_child(section.section_path) + ] + if result: + return result + return [ + ParsedSubsection( + title=section.section_title, + content=section.content, + path=section.section_path, + ) + for section in sections + if section.section_title.strip().lower() != "summary" + ] + + def _is_details_child(self, section_path: str) -> bool: + parts = [item.strip() for item in section_path.split(" > ") if item.strip()] + for index, part in enumerate(parts): + if part.lower() == "details" and len(parts) == index + 2: + return True + return False + + def _title_from_body(self, body: str) -> str: + match = re.search(r"^#\s+(.+)$", body, flags=re.MULTILINE) + return match.group(1).strip() if match else "" diff --git a/src/app/core/rag/indexing/docs/document_builder.py b/src/app/core/rag/indexing/docs/document_builder.py new file mode 100644 index 0000000..3884421 --- /dev/null +++ b/src/app/core/rag/indexing/docs/document_builder.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +from app.core.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource +from app.core.rag.indexing.docs.chunkers.markdown_chunker import SectionChunk +from app.core.rag.indexing.docs.frontmatter_view import DocsFrontmatterView + + +class DocsDocumentBuilder: + def build_document_catalog(self, source: RagSource, frontmatter: dict, summary_text: str, doc_kind: str, *, fallback_title: str) -> RagDocument: + view = DocsFrontmatterView(frontmatter) + document_id = view.document_id or source.path + metadata = { + "document_id": document_id, + "type": view.doc_type, + "name": view.name, + "title": view.title(fallback_title), + "module": view.module, + "domain": view.domain, + "subdomain": view.subdomain, + "layer": view.layer, + "status": view.status, + "updated_at": view.updated_at, + "tags": view.tags, + "entities": view.entities, + "parent": view.parent, + "children": view.children, + "links": view.links, + "source_path": source.path, + "summary_text": summary_text[:4000], + "doc_kind": doc_kind, + "artifact_type": "DOCS", + } + return RagDocument( + layer=RagLayer.DOCS_DOCUMENT_CATALOG, + source=source, + title=metadata["title"] or document_id, + text=summary_text[:4000] or metadata["title"] or document_id, + metadata=metadata, + ) + + def build_doc_chunk(self, source: RagSource, chunk: SectionChunk, frontmatter: dict, doc_kind: str) -> RagDocument: + view = DocsFrontmatterView(frontmatter) + document_id = view.document_id or source.path + metadata = { + "document_id": document_id, + "type": view.doc_type, + "module": view.module, + "domain": view.domain, + "subdomain": view.subdomain, + "tags": view.tags, + "section_path": chunk.section_path, + "section_title": chunk.section_title, + "order": chunk.order, + "doc_kind": doc_kind, + "source_path": source.path, + "artifact_type": "DOCS", + } + return RagDocument( + layer=RagLayer.DOCS_DOC_CHUNKS, + source=source, + title=f"{document_id}:{chunk.section_title}", + text=chunk.content, + metadata=metadata, + links=[EvidenceLink(type=EvidenceType.DOC_SECTION, target_id=document_id, path=source.path, note=chunk.section_path)], + ) + + def build_entity_record(self, source: RagSource, frontmatter: dict, entity: str) -> RagDocument: + view = DocsFrontmatterView(frontmatter) + document_id = view.document_id or source.path + metadata = { + "entity_name": entity, + "document_id": document_id, + "document_type": view.doc_type, + "module": view.module, + "domain": view.domain, + "subdomain": view.subdomain, + "tags": view.tags, + "source_path": source.path, + "artifact_type": "DOCS", + } + return RagDocument( + layer=RagLayer.DOCS_ENTITY_CATALOG, + source=source, + title=entity, + text=entity, + metadata=metadata, + ) diff --git a/src/app/modules/rag/indexing/docs/entity_extractor.py b/src/app/core/rag/indexing/docs/entity_extractor.py similarity index 100% rename from src/app/modules/rag/indexing/docs/entity_extractor.py rename to src/app/core/rag/indexing/docs/entity_extractor.py diff --git a/src/app/modules/rag/indexing/docs/fact_extractor.py b/src/app/core/rag/indexing/docs/fact_extractor.py similarity index 100% rename from src/app/modules/rag/indexing/docs/fact_extractor.py rename to src/app/core/rag/indexing/docs/fact_extractor.py diff --git a/src/app/modules/rag/indexing/docs/file_filter.py b/src/app/core/rag/indexing/docs/file_filter.py similarity index 92% rename from src/app/modules/rag/indexing/docs/file_filter.py rename to src/app/core/rag/indexing/docs/file_filter.py index 5fc7b2e..8eeaae0 100644 --- a/src/app/modules/rag/indexing/docs/file_filter.py +++ b/src/app/core/rag/indexing/docs/file_filter.py @@ -2,7 +2,7 @@ from __future__ import annotations from pathlib import PurePosixPath -from app.modules.rag.indexing.common.path_filter import should_skip_indexing_path +from app.core.rag.indexing.common.path_filter import should_skip_indexing_path class DocsFileFilter: diff --git a/src/app/modules/rag/indexing/docs/frontmatter_parser.py b/src/app/core/rag/indexing/docs/frontmatter_parser.py similarity index 100% rename from src/app/modules/rag/indexing/docs/frontmatter_parser.py rename to src/app/core/rag/indexing/docs/frontmatter_parser.py diff --git a/src/app/core/rag/indexing/docs/frontmatter_view.py b/src/app/core/rag/indexing/docs/frontmatter_view.py new file mode 100644 index 0000000..dee5b36 --- /dev/null +++ b/src/app/core/rag/indexing/docs/frontmatter_view.py @@ -0,0 +1,85 @@ +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(slots=True, frozen=True) +class DocsFrontmatterView: + raw: dict + + @property + def document_id(self) -> str: + return str(self.raw.get("id") or "").strip() + + @property + def doc_type(self) -> str: + return str(self.raw.get("type") or self.raw.get("doc_type") or "").strip() + + @property + def name(self) -> str: + return str(self.raw.get("name") or "").strip() + + @property + def module(self) -> str: + return str(self.raw.get("module") or "").strip() + + @property + def domain(self) -> str: + return str(self.raw.get("domain") or "").strip() + + @property + def subdomain(self) -> str: + return str(self.raw.get("subdomain") or self.raw.get("sub_domain") or "").strip() + + @property + def layer(self) -> str: + return str(self.raw.get("layer") or "").strip() + + @property + def status(self): + return self.raw.get("status") + + @property + def updated_at(self): + return self.raw.get("updated_at") + + @property + def tags(self) -> list[str]: + return self._string_list(self.raw.get("tags")) + + @property + def entities(self) -> list[str]: + return self._string_list(self.raw.get("entities")) + + @property + def related_docs(self) -> list[str]: + return self._string_list(self.raw.get("related_docs")) + + @property + def parent(self) -> str | None: + value = str(self.raw.get("parent") or "").strip() + return value or None + + @property + def children(self) -> list[str]: + return self._string_list(self.raw.get("children")) + + @property + def links(self): + value = self.raw.get("links") + if isinstance(value, (dict, list)): + return value + return {} + + def title(self, fallback: str = "") -> str: + return str(self.raw.get("title") or "").strip() or fallback + + def _string_list(self, value) -> list[str]: + if not isinstance(value, list): + return [] + result: list[str] = [] + for item in value: + normalized = str(item or "").strip() + if normalized and normalized not in result: + result.append(normalized) + return result diff --git a/src/app/core/rag/indexing/docs/integration_extractor.py b/src/app/core/rag/indexing/docs/integration_extractor.py new file mode 100644 index 0000000..296fe51 --- /dev/null +++ b/src/app/core/rag/indexing/docs/integration_extractor.py @@ -0,0 +1,93 @@ +from __future__ import annotations + +import yaml + +from app.core.rag.indexing.docs.chunkers.markdown_chunker import SectionChunk +from app.core.rag.indexing.docs.models import IntegrationRecord + + +class DocsIntegrationExtractor: + _SECTION_TITLES = {"integrations", "интеграции"} + + def extract(self, sections: list[SectionChunk]) -> list[IntegrationRecord]: + records: list[IntegrationRecord] = [] + for section in sections: + if not self._is_integration_section(section.section_path): + continue + payload = self._payload(section.content) + target = str(payload.get("target") or "").strip() + if not target: + continue + details = payload.get("details") + if not isinstance(details, dict): + details = {} + records.append( + IntegrationRecord( + name=section.section_title, + target=target, + target_type=str(payload.get("target_type") or "").strip(), + direction=str(payload.get("direction") or "").strip(), + interaction=str(payload.get("interaction") or "").strip(), + via=str(payload.get("via") or "").strip(), + purpose=str(payload.get("purpose") or "").strip(), + details=details, + anchor=section.section_path, + ) + ) + return records + + def _is_integration_section(self, section_path: str) -> bool: + parts = [item.strip().lower() for item in section_path.split(" > ") if item.strip()] + return any(part in self._SECTION_TITLES for part in parts[:-1]) or (parts and parts[-1] in self._SECTION_TITLES) + + def _payload(self, text: str) -> dict: + payload: dict = {} + details_lines: list[str] = [] + collecting_details = False + for raw_line in text.splitlines(): + if not raw_line.strip(): + continue + indent = len(raw_line) - len(raw_line.lstrip(" ")) + stripped = raw_line.strip() + if indent > 0 and collecting_details: + details_lines.append(raw_line) + continue + if not stripped.startswith("- "): + continue + key, value = self._split_key_value(stripped[2:]) + if not key: + continue + if key == "details": + collecting_details = True + details_lines = [] + if value: + payload[key] = self._yaml_value(value) + continue + collecting_details = False + payload[key] = self._yaml_value(value) + if details_lines: + payload["details"] = self._details_payload(details_lines) + return payload + + def _details_payload(self, lines: list[str]) -> dict: + normalized: list[str] = [] + for raw_line in lines: + line = raw_line[2:] if raw_line.startswith(" ") else raw_line + stripped = line.lstrip() + indent = len(line) - len(stripped) + if indent == 0 and stripped.startswith("- "): + stripped = stripped[2:] + normalized.append((" " * indent) + stripped) + payload = yaml.safe_load("\n".join(normalized)) or {} + return payload if isinstance(payload, dict) else {} + + def _split_key_value(self, text: str) -> tuple[str, str]: + if ":" not in text: + return text.strip(), "" + key, value = text.split(":", 1) + return key.strip(), value.strip() + + def _yaml_value(self, value: str): + if not value: + return "" + return yaml.safe_load(value) diff --git a/src/app/modules/rag/indexing/docs/models.py b/src/app/core/rag/indexing/docs/models.py similarity index 78% rename from src/app/modules/rag/indexing/docs/models.py rename to src/app/core/rag/indexing/docs/models.py index 2e5efd0..5c523c5 100644 --- a/src/app/modules/rag/indexing/docs/models.py +++ b/src/app/core/rag/indexing/docs/models.py @@ -13,6 +13,7 @@ class ParsedTopLevelSection: class ParsedSubsection: title: str content: str + path: str @dataclass(slots=True) @@ -37,6 +38,19 @@ class WorkflowScenario: anchor: str +@dataclass(slots=True) +class IntegrationRecord: + name: str + target: str + target_type: str + direction: str + interaction: str + via: str + purpose: str + details: dict + anchor: str + + @dataclass(slots=True) class RelationEdge: source_id: str diff --git a/src/app/modules/rag/indexing/docs/pipeline.py b/src/app/core/rag/indexing/docs/pipeline.py similarity index 52% rename from src/app/modules/rag/indexing/docs/pipeline.py rename to src/app/core/rag/indexing/docs/pipeline.py index aa5606c..291326e 100644 --- a/src/app/modules/rag/indexing/docs/pipeline.py +++ b/src/app/core/rag/indexing/docs/pipeline.py @@ -1,16 +1,19 @@ from __future__ import annotations -from app.modules.rag.contracts import RagDocument, RagSource -from app.modules.rag.indexing.docs.chunkers.markdown_chunker import MarkdownDocChunker -from app.modules.rag.indexing.docs.classifier import DocsClassifier -from app.modules.rag.indexing.docs.content_parser import DocsContentParser -from app.modules.rag.indexing.docs.document_builder import DocsDocumentBuilder -from app.modules.rag.indexing.docs.entity_extractor import DocsEntityExtractor -from app.modules.rag.indexing.docs.fact_extractor import DocsFactExtractor -from app.modules.rag.indexing.docs.file_filter import DocsFileFilter -from app.modules.rag.indexing.docs.frontmatter_parser import DocsFrontmatterParser -from app.modules.rag.indexing.docs.relation_extractor import DocsRelationExtractor -from app.modules.rag.indexing.docs.workflow_extractor import DocsWorkflowExtractor +from app.core.rag.contracts import RagDocument, RagSource +from app.core.rag.indexing.docs.chunkers.markdown_chunker import MarkdownDocChunker +from app.core.rag.indexing.docs.classifier import DocsClassifier +from app.core.rag.indexing.docs.content_parser import DocsContentParser +from app.core.rag.indexing.docs.document_builder import DocsDocumentBuilder +from app.core.rag.indexing.docs.entity_extractor import DocsEntityExtractor +from app.core.rag.indexing.docs.fact_extractor import DocsFactExtractor +from app.core.rag.indexing.docs.file_filter import DocsFileFilter +from app.core.rag.indexing.docs.frontmatter_parser import DocsFrontmatterParser +from app.core.rag.indexing.docs.frontmatter_view import DocsFrontmatterView +from app.core.rag.indexing.docs.integration_extractor import DocsIntegrationExtractor +from app.core.rag.indexing.docs.relation_extractor import DocsRelationExtractor +from app.core.rag.indexing.docs.support_layer_builder import DocsSupportLayerBuilder +from app.core.rag.indexing.docs.workflow_extractor import DocsWorkflowExtractor class DocsIndexingPipeline: @@ -19,12 +22,14 @@ class DocsIndexingPipeline: self._classifier = DocsClassifier() self._chunker = MarkdownDocChunker() self._builder = DocsDocumentBuilder() + self._support_builder = DocsSupportLayerBuilder() self._frontmatter = DocsFrontmatterParser() self._content = DocsContentParser() self._facts = DocsFactExtractor() self._entities = DocsEntityExtractor() self._workflows = DocsWorkflowExtractor() self._relations = DocsRelationExtractor() + self._integrations = DocsIntegrationExtractor() def supports(self, path: str) -> bool: return self._filter.should_index(path) @@ -33,8 +38,14 @@ class DocsIndexingPipeline: source = RagSource(repo_id=repo_id, commit_sha=commit_sha, path=path) frontmatter, body = self._frontmatter.split(content) doc_kind = self._classifier.classify(path) - parsed = self._content.parse(frontmatter, body, fallback_title=self._fallback_title(path)) - sections = self._chunker.chunk(parsed.body or body) + sections = self._chunker.chunk(body) + parsed = self._content.parse( + frontmatter, + body, + fallback_title=self._fallback_title(path), + sections=sections, + ) + frontmatter_view = DocsFrontmatterView(parsed.frontmatter) docs: list[RagDocument] = [] docs.append( self._builder.build_document_catalog( @@ -47,24 +58,28 @@ class DocsIndexingPipeline: ) for section in sections: docs.append(self._builder.build_doc_chunk(source, section, parsed.frontmatter, doc_kind)) - document_id = str(parsed.frontmatter.get("id") or source.path) + document_id = frontmatter_view.document_id or source.path for fact in self._facts.extract(parsed.frontmatter, sections): docs.append( - self._builder.build_fact( + self._support_builder.build_fact( source, subject_id=fact["subject_id"] or document_id, predicate=fact["predicate"], obj=fact["object"], object_ref=fact.get("object_ref"), anchor=fact["anchor"], + domain=frontmatter_view.domain, + subdomain=frontmatter_view.subdomain, ) ) for entity in self._entities.extract(parsed.frontmatter): docs.append(self._builder.build_entity_record(source, parsed.frontmatter, entity)) for workflow in self._workflows.extract(parsed.detail_sections): - docs.append(self._builder.build_workflow_record(source, parsed.frontmatter, workflow)) + docs.append(self._support_builder.build_workflow_record(source, parsed.frontmatter, workflow)) for edge in self._relations.extract(parsed.frontmatter, source_id=document_id): - docs.append(self._builder.build_relation_record(source, edge)) + docs.append(self._support_builder.build_relation_record(source, parsed.frontmatter, edge)) + for integration in self._integrations.extract(sections): + docs.append(self._support_builder.build_integration_record(source, parsed.frontmatter, integration)) return docs def _fallback_title(self, path: str) -> str: diff --git a/src/app/core/rag/indexing/docs/relation_extractor.py b/src/app/core/rag/indexing/docs/relation_extractor.py new file mode 100644 index 0000000..c38835f --- /dev/null +++ b/src/app/core/rag/indexing/docs/relation_extractor.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +from app.core.rag.indexing.docs.frontmatter_view import DocsFrontmatterView +from app.core.rag.indexing.docs.models import RelationEdge + + +class DocsRelationExtractor: + def extract(self, frontmatter: dict, *, source_id: str) -> list[RelationEdge]: + edges: list[RelationEdge] = [] + view = DocsFrontmatterView(frontmatter) + if view.parent: + edges.append( + RelationEdge( + source_id=source_id, + relation_type="parent", + target_id=view.parent, + anchor="frontmatter.parent", + ) + ) + for child in view.children: + edges.append( + RelationEdge( + source_id=source_id, + relation_type="child", + target_id=child, + anchor="frontmatter.children", + ) + ) + for related_doc in view.related_docs: + edges.append( + RelationEdge( + source_id=source_id, + relation_type="related_doc", + target_id=related_doc, + anchor="frontmatter.related_docs", + ) + ) + for entity in view.entities: + edges.append( + RelationEdge( + source_id=source_id, + relation_type="mentions_entity", + target_id=entity, + anchor="frontmatter.entities", + ) + ) + links = view.links + if isinstance(links, dict): + for relation_type, values in links.items(): + for target in self._targets(values): + edges.append( + RelationEdge( + source_id=source_id, + relation_type=str(relation_type or "").strip(), + target_id=target, + anchor=f"frontmatter.links.{relation_type}", + ) + ) + return edges + for item in links if isinstance(links, list) else []: + if not isinstance(item, dict): + continue + relation_type = str(item.get("type") or "").strip() + target = str(item.get("target") or "").strip() + if relation_type and target: + edges.append( + RelationEdge( + source_id=source_id, + relation_type=relation_type, + target_id=target, + anchor="frontmatter.links", + ) + ) + return edges + + def _targets(self, value) -> list[str]: + if not isinstance(value, list): + return [] + result: list[str] = [] + for item in value: + if isinstance(item, dict): + target = str(item.get("target") or item.get("id") or "").strip() + else: + target = str(item or "").strip() + if target: + result.append(target) + return result diff --git a/src/app/core/rag/indexing/docs/support_layer_builder.py b/src/app/core/rag/indexing/docs/support_layer_builder.py new file mode 100644 index 0000000..0ba956a --- /dev/null +++ b/src/app/core/rag/indexing/docs/support_layer_builder.py @@ -0,0 +1,139 @@ +from __future__ import annotations + +from hashlib import sha256 + +from app.core.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource +from app.core.rag.indexing.docs.frontmatter_view import DocsFrontmatterView +from app.core.rag.indexing.docs.models import IntegrationRecord, RelationEdge, WorkflowScenario + + +class DocsSupportLayerBuilder: + def build_fact( + self, + source: RagSource, + *, + subject_id: str, + predicate: str, + obj: str, + object_ref: str | None, + anchor: str, + tags: list[str] | None = None, + domain: str = "", + subdomain: str = "", + ) -> RagDocument: + fact_id = sha256(f"{subject_id}|{predicate}|{obj}|{source.path}|{anchor}".encode("utf-8")).hexdigest() + metadata = { + "fact_id": fact_id, + "subject_id": subject_id, + "predicate": predicate, + "object": obj, + "object_ref": object_ref, + "anchor": anchor, + "tags": tags or [], + "domain": domain, + "subdomain": subdomain, + "source_path": source.path, + "artifact_type": "DOCS", + } + return RagDocument( + layer=RagLayer.DOCS_FACT_INDEX, + source=source, + title=f"{subject_id}:{predicate}", + text=f"{subject_id} {predicate} {obj}".strip(), + metadata=metadata, + links=[EvidenceLink(type=EvidenceType.DOC_FACT, target_id=fact_id, path=source.path, note=anchor)], + ) + + def build_workflow_record(self, source: RagSource, frontmatter: dict, workflow: WorkflowScenario) -> RagDocument: + view = DocsFrontmatterView(frontmatter) + document_id = view.document_id or source.path + workflow_id = sha256(f"{document_id}|workflow|{workflow.name}|{workflow.anchor}".encode("utf-8")).hexdigest() + metadata = { + "workflow_id": workflow_id, + "document_id": document_id, + "workflow_name": workflow.name, + "preconditions": workflow.preconditions, + "trigger": workflow.trigger, + "main_flow": workflow.main_flow, + "alternative_flow": workflow.alternative_flow, + "error_handling": workflow.error_handling, + "postconditions": workflow.postconditions, + "domain": view.domain, + "subdomain": view.subdomain, + "source_path": source.path, + "artifact_type": "DOCS", + } + text = "\n".join( + [ + workflow.name, + *workflow.preconditions, + *workflow.trigger, + *workflow.main_flow, + *workflow.alternative_flow, + *workflow.error_handling, + *workflow.postconditions, + ] + ).strip() + return RagDocument( + layer=RagLayer.DOCS_WORKFLOW_INDEX, + source=source, + title=workflow.name, + text=text[:4000], + metadata=metadata, + ) + + def build_relation_record(self, source: RagSource, frontmatter: dict, edge: RelationEdge) -> RagDocument: + view = DocsFrontmatterView(frontmatter) + relation_id = sha256(f"{edge.source_id}|{edge.relation_type}|{edge.target_id}|{source.path}|{edge.anchor}".encode("utf-8")).hexdigest() + metadata = { + "relation_id": relation_id, + "source_id": edge.source_id, + "relation_type": edge.relation_type, + "target_id": edge.target_id, + "anchor": edge.anchor, + "domain": view.domain, + "subdomain": view.subdomain, + "source_path": source.path, + "artifact_type": "DOCS", + } + return RagDocument( + layer=RagLayer.DOCS_RELATION_GRAPH, + source=source, + title=f"{edge.source_id}:{edge.relation_type}", + text=f"{edge.source_id} {edge.relation_type} {edge.target_id}", + metadata=metadata, + ) + + def build_integration_record(self, source: RagSource, frontmatter: dict, integration: IntegrationRecord) -> RagDocument: + view = DocsFrontmatterView(frontmatter) + source_id = view.document_id or source.path + integration_id = sha256(f"{source_id}|{integration.target}|{integration.anchor}".encode("utf-8")).hexdigest() + metadata = { + "integration_id": integration_id, + "source_id": source_id, + "source_type": view.doc_type, + "target": integration.target, + "target_type": integration.target_type, + "direction": integration.direction, + "interaction": integration.interaction, + "via": integration.via, + "purpose": integration.purpose, + "details": integration.details, + "domain": view.domain, + "subdomain": view.subdomain, + "source_path": source.path, + "anchor": integration.anchor, + "artifact_type": "DOCS", + } + text = " | ".join( + part + for part in [integration.name, integration.target, integration.interaction, integration.via, integration.purpose] + if part + ) + return RagDocument( + layer=RagLayer.DOCS_INTEGRATION_INDEX, + source=source, + title=integration.name, + text=text[:4000] or integration.target, + metadata=metadata, + ) diff --git a/src/app/modules/rag/indexing/docs/workflow_extractor.py b/src/app/core/rag/indexing/docs/workflow_extractor.py similarity index 80% rename from src/app/modules/rag/indexing/docs/workflow_extractor.py rename to src/app/core/rag/indexing/docs/workflow_extractor.py index 6076337..0cbb736 100644 --- a/src/app/modules/rag/indexing/docs/workflow_extractor.py +++ b/src/app/core/rag/indexing/docs/workflow_extractor.py @@ -1,9 +1,12 @@ from __future__ import annotations -from app.modules.rag.indexing.docs.models import ParsedSubsection, WorkflowScenario +import re + +from app.core.rag.indexing.docs.models import ParsedSubsection, WorkflowScenario class DocsWorkflowExtractor: + _SECTION_TITLES = {"сценарий", "технический use case"} _LABELS = { "название": "name", "предусловия": "preconditions", @@ -15,7 +18,10 @@ class DocsWorkflowExtractor: } def extract(self, sections: list[ParsedSubsection]) -> list[WorkflowScenario]: - scenario_section = next((section for section in sections if section.title.lower() == "сценарий"), None) + scenario_section = next( + (section for section in sections if section.title.strip().lower() in self._SECTION_TITLES), + None, + ) if scenario_section is None or not scenario_section.content.strip(): return [] fields = self._fields(scenario_section.content) @@ -29,7 +35,7 @@ class DocsWorkflowExtractor: alternative_flow=fields.get("alternative_flow", []), error_handling=fields.get("error_handling", []), postconditions=fields.get("postconditions", []), - anchor="Details/Сценарий", + anchor=scenario_section.path, ) ] @@ -51,8 +57,8 @@ class DocsWorkflowExtractor: normalized = line if normalized.startswith("- "): normalized = normalized[2:].strip() - elif normalized[:2].isdigit() and normalized[1:3] == ". ": - normalized = normalized[3:].strip() + elif re.match(r"^\d+\.\s+", normalized): + normalized = re.sub(r"^\d+\.\s+", "", normalized).strip() if normalized: fields[current_key].append(normalized) return fields diff --git a/src/app/modules/rag/job_store.py b/src/app/core/rag/indexing/job_store.py similarity index 97% rename from src/app/modules/rag/job_store.py rename to src/app/core/rag/indexing/job_store.py index bbe2b90..61480e4 100644 --- a/src/app/modules/rag/job_store.py +++ b/src/app/core/rag/indexing/job_store.py @@ -3,7 +3,7 @@ from dataclasses import dataclass from uuid import uuid4 -from app.modules.rag.persistence.repository import RagRepository +from app.core.rag.persistence.repository import RagRepository from app.schemas.common import ErrorPayload, ModuleName from app.schemas.indexing import IndexJobStatus diff --git a/src/app/modules/rag/indexing_service.py b/src/app/core/rag/indexing/orchestrator.py similarity index 95% rename from src/app/modules/rag/indexing_service.py rename to src/app/core/rag/indexing/orchestrator.py index e3adc07..85bdf73 100644 --- a/src/app/modules/rag/indexing_service.py +++ b/src/app/core/rag/indexing/orchestrator.py @@ -5,15 +5,15 @@ from collections import defaultdict from app.schemas.common import ErrorPayload, ModuleName from app.schemas.indexing import IndexJobStatus -from app.modules.contracts import RagIndexer -from app.modules.rag.indexing.common.path_filter import ( +from app.core.contracts import RagIndexer +from app.core.rag.indexing.common.path_filter import ( count_indexable_change_upserts, filter_changes_for_indexing, filter_snapshot_files, ) -from app.modules.rag.job_store import IndexJob, IndexJobStore -from app.modules.shared.event_bus import EventBus -from app.modules.shared.retry_executor import RetryExecutor +from app.core.rag.indexing.job_store import IndexJob, IndexJobStore +from app.core.shared.messaging import EventBus +from app.core.shared.resilience import RetryExecutor class IndexingOrchestrator: diff --git a/src/app/modules/rag/services/rag_service.py b/src/app/core/rag/indexing/service.py similarity index 93% rename from src/app/modules/rag/services/rag_service.py rename to src/app/core/rag/indexing/service.py index 43802fe..e366ab9 100644 --- a/src/app/modules/rag/services/rag_service.py +++ b/src/app/core/rag/indexing/service.py @@ -8,17 +8,17 @@ from collections.abc import Awaitable, Callable from inspect import isawaitable from typing import TYPE_CHECKING -from app.modules.rag.contracts import RagDocument -from app.modules.rag.indexing.common.path_filter import filter_snapshot_files -from app.modules.rag.indexing.code.pipeline import CodeIndexingPipeline -from app.modules.rag.indexing.common.report import IndexReport -from app.modules.rag.indexing.docs.pipeline import DocsIndexingPipeline -from app.modules.rag.embedding.gigachat_embedder import GigaChatEmbedder +from app.core.rag.contracts import RagDocument +from app.core.rag.indexing.common.path_filter import filter_snapshot_files +from app.core.rag.indexing.code.pipeline import CodeIndexingPipeline +from app.core.rag.indexing.common.report import IndexReport +from app.core.rag.indexing.docs.pipeline import DocsIndexingPipeline +from app.core.rag.embedding.gigachat_embedder import GigaChatEmbedder LOGGER = logging.getLogger(__name__) if TYPE_CHECKING: - from app.modules.rag.persistence.repository import RagRepository + from app.core.rag.persistence.repository import RagRepository class RagService: @@ -157,7 +157,7 @@ class RagService: return docs def _document_metadata(self, file: dict, repo_id: str, blob_sha: str) -> dict: - return { + metadata = { "blob_sha": blob_sha, "repo_id": repo_id, "artifact_type": file.get("artifact_type"), @@ -169,6 +169,7 @@ class RagService: "last_modified": file.get("last_modified"), "staleness_score": file.get("staleness_score"), } + return {key: value for key, value in metadata.items() if value not in (None, "", [], {})} def _resolve_repo_id(self, rag_session_id: str) -> str: session = self._repo.get_session(rag_session_id) diff --git a/src/app/core/rag/ingest_api.py b/src/app/core/rag/ingest_api.py new file mode 100644 index 0000000..c770427 --- /dev/null +++ b/src/app/core/rag/ingest_api.py @@ -0,0 +1,11 @@ +"""Стандартный API ingest для пакета RAG. + +Индексация документов выполняется через :class:`RagService` (``index_snapshot``, +``index_changes``). В приложении экземпляр доступен как ``RagModule.rag``. +""" + +from __future__ import annotations + +from app.core.rag.indexing.service import RagService + +__all__ = ["RagService"] diff --git a/src/app/modules/rag/intent_router.md b/src/app/core/rag/intent_router.md similarity index 100% rename from src/app/modules/rag/intent_router.md rename to src/app/core/rag/intent_router.md diff --git a/src/app/core/rag/module.py b/src/app/core/rag/module.py new file mode 100644 index 0000000..90e655b --- /dev/null +++ b/src/app/core/rag/module.py @@ -0,0 +1,56 @@ +"""RAG module composition and application-facing operations.""" + +from __future__ import annotations + +from app.infra.exceptions import AppError +from app.core.rag.embedding.gigachat_embedder import GigaChatEmbedder +from app.core.rag.indexing import IndexJob, IndexJobStore, IndexingOrchestrator, RagService +from app.core.rag.persistence import RagRepository, RagSession, RagSessionStore +from app.core.rag.retrieval import TextChunker +from app.core.shared.gigachat.client import GigaChatClient +from app.core.shared.gigachat.settings import GigaChatSettings +from app.core.shared.gigachat.token_provider import GigaChatTokenProvider +from app.core.shared.messaging import EventBus +from app.core.shared.resilience import RetryExecutor +from app.schemas.common import ModuleName + + +class RagModule: + def __init__(self, event_bus: EventBus, retry: RetryExecutor, repository: RagRepository, ensure_ready=None) -> None: + self._events = event_bus + self._ensure_ready = ensure_ready or (lambda: None) + settings = GigaChatSettings.from_env() + token_provider = GigaChatTokenProvider(settings) + client = GigaChatClient(settings, token_provider) + embedder = GigaChatEmbedder(client) + self.repository = repository + self.rag = RagService(embedder=embedder, repository=repository, chunker=TextChunker()) + self.sessions = RagSessionStore(repository) + self.jobs = IndexJobStore(repository) + self.indexing = IndexingOrchestrator(store=self.jobs, rag=self.rag, events=event_bus, retry=retry) + + async def create_session(self, project_id: str, files: list[dict]) -> tuple[RagSession, IndexJob]: + self._ensure_ready() + session = self.sessions.create(project_id) + job = await self.indexing.enqueue_snapshot(rag_session_id=session.rag_session_id, files=files) + return session, job + + def get_session_job(self, rag_session_id: str, index_job_id: str) -> IndexJob: + self._ensure_ready() + job = self._require_job(index_job_id) + if job.rag_session_id != rag_session_id: + raise AppError("not_found", f"Index job not found: {index_job_id}", ModuleName.RAG) + return job + + async def subscribe_session_job_events(self, rag_session_id: str, index_job_id: str): + job = self.get_session_job(rag_session_id, index_job_id) + return job.index_job_id, await self._events.subscribe(job.index_job_id, replay=True) + + async def unsubscribe_job_events(self, index_job_id: str, queue) -> None: + await self._events.unsubscribe(index_job_id, queue) + + def _require_job(self, index_job_id: str) -> IndexJob: + job = self.jobs.get(index_job_id) + if not job: + raise AppError("not_found", f"Index job not found: {index_job_id}", ModuleName.RAG) + return job diff --git a/src/app/core/rag/persistence/__init__.py b/src/app/core/rag/persistence/__init__.py new file mode 100644 index 0000000..84173bc --- /dev/null +++ b/src/app/core/rag/persistence/__init__.py @@ -0,0 +1,17 @@ +__all__ = ["RagRepository", "RagSession", "RagSessionStore"] + + +def __getattr__(name: str): + if name == "RagRepository": + from app.core.rag.persistence.repository import RagRepository + + return RagRepository + if name == "RagSession": + from app.core.rag.persistence.session_store import RagSession + + return RagSession + if name == "RagSessionStore": + from app.core.rag.persistence.session_store import RagSessionStore + + return RagSessionStore + raise AttributeError(name) diff --git a/src/app/modules/rag/persistence/cache_repository.py b/src/app/core/rag/persistence/cache_repository.py similarity index 78% rename from src/app/modules/rag/persistence/cache_repository.py rename to src/app/core/rag/persistence/cache_repository.py index 5987c08..0f90af2 100644 --- a/src/app/modules/rag/persistence/cache_repository.py +++ b/src/app/core/rag/persistence/cache_repository.py @@ -5,8 +5,7 @@ from datetime import date, datetime from sqlalchemy import text -from app.modules.rag.contracts import EvidenceLink, RagDocument, RagSource, RagSpan -from app.modules.rag.contracts.enums import RagLayer +from app.core.rag.contracts import EvidenceLink, RagDocument, RagSource, RagSpan class RagCacheRepository: @@ -27,24 +26,6 @@ class RagCacheRepository: conn.execute(text(self._insert_sql()), self._insert_params(repo_id, path, blob_sha, idx, doc)) conn.commit() - def record_repo_cache(self, *, project_id: str, commit_sha: str | None, changed_files: list[str], summary: str) -> None: - docs: list[RagDocument] = [] - for idx, path in enumerate(changed_files): - docs.append( - RagDocument( - layer=RagLayer.CODE_SOURCE_CHUNKS, - lang="python" if path.endswith(".py") else None, - source=RagSource(project_id, commit_sha, path), - title=path, - text=f"repo_webhook:{path}:{summary[:300]}", - metadata={"chunk_index": idx, "artifact_type": "CODE", "section": "repo_webhook"}, - ) - ) - for doc in docs: - blob_sha = self._blob_sha(commit_sha, doc.source.path) - doc.metadata["blob_sha"] = blob_sha - self.cache_documents(project_id, doc.source.path, blob_sha, [doc]) - def _select_sql(self) -> str: return """ SELECT layer, lang, path, title, content, metadata_json, links_json, span_start, span_end, @@ -101,13 +82,8 @@ class RagCacheRepository: embedding=self._parse_vector(str(row["embedding_txt"] or "")), ) - def _blob_sha(self, commit_sha: str | None, path: str) -> str: - from hashlib import sha256 - - return sha256(f"{commit_sha or 'no-commit'}:{path}".encode("utf-8")).hexdigest() - def _engine(self): - from app.modules.shared.db import get_engine + from app.core.shared.database import get_engine return get_engine() diff --git a/src/app/modules/rag/persistence/document_repository.py b/src/app/core/rag/persistence/document_repository.py similarity index 98% rename from src/app/modules/rag/persistence/document_repository.py rename to src/app/core/rag/persistence/document_repository.py index ee79307..e7c0993 100644 --- a/src/app/modules/rag/persistence/document_repository.py +++ b/src/app/core/rag/persistence/document_repository.py @@ -5,7 +5,7 @@ from datetime import date, datetime from sqlalchemy import text -from app.modules.rag.contracts import RagDocument +from app.core.rag.contracts import RagDocument class RagDocumentRepository: diff --git a/src/app/modules/rag/persistence/job_repository.py b/src/app/core/rag/persistence/job_repository.py similarity index 98% rename from src/app/modules/rag/persistence/job_repository.py rename to src/app/core/rag/persistence/job_repository.py index 72fba06..ad20ceb 100644 --- a/src/app/modules/rag/persistence/job_repository.py +++ b/src/app/core/rag/persistence/job_repository.py @@ -4,7 +4,7 @@ from dataclasses import dataclass from sqlalchemy import text -from app.modules.shared.db import get_engine +from app.core.shared.database import get_engine @dataclass diff --git a/src/app/modules/rag/persistence/query_repository.py b/src/app/core/rag/persistence/query_repository.py similarity index 55% rename from src/app/modules/rag/persistence/query_repository.py rename to src/app/core/rag/persistence/query_repository.py index 8a020b2..593f7ed 100644 --- a/src/app/modules/rag/persistence/query_repository.py +++ b/src/app/core/rag/persistence/query_repository.py @@ -4,8 +4,8 @@ import json from sqlalchemy import bindparam, text -from app.modules.rag.persistence.retrieval_statement_builder import RetrievalStatementBuilder -from app.modules.shared.db import get_engine +from app.core.rag.persistence.retrieval_statement_builder import RetrievalStatementBuilder +from app.core.shared.database import get_engine class RagQueryRepository: @@ -135,6 +135,105 @@ class RagQueryRepository: rows = conn.execute(stmt, params).mappings().fetchall() return [self._row_to_dict(row) for row in rows] + def retrieve_by_document_ids( + self, + rag_session_id: str, + *, + document_ids: list[str], + layers: list[str] | None = None, + limit: int = 200, + ) -> list[dict]: + normalized_ids = [str(item).strip() for item in document_ids if str(item).strip()] + if not normalized_ids: + return [] + params: dict = { + "sid": rag_session_id, + "lim": max(1, int(limit)), + "doc_ids_document": normalized_ids, + "doc_ids_doc": normalized_ids, + "doc_ids_subject": normalized_ids, + } + stmt = text( + """ + SELECT path, content, layer, title, metadata_json, span_start, span_end, + 0 AS lexical_rank, + 0 AS prefer_bonus, + 0 AS test_penalty, + 0 AS structural_rank, + 0 AS layer_rank, + 0 AS distance + FROM rag_chunks + WHERE rag_session_id = :sid + AND ( + COALESCE(metadata_json->>'document_id', '') IN :doc_ids_document + OR COALESCE(metadata_json->>'doc_id', '') IN :doc_ids_doc + OR COALESCE(metadata_json->>'subject_id', '') IN :doc_ids_subject + ) + ORDER BY + CASE layer + WHEN 'D1_DOCUMENT_CATALOG' THEN 0 + WHEN 'D2_FACT_INDEX' THEN 1 + WHEN 'D0_DOC_CHUNKS' THEN 2 + WHEN 'D4_WORKFLOW_INDEX' THEN 3 + WHEN 'D6_INTEGRATION_INDEX' THEN 4 + WHEN 'D5_RELATION_GRAPH' THEN 5 + ELSE 9 + END, + path ASC, + COALESCE(span_start, 0) ASC, + COALESCE(chunk_index, 0) ASC + LIMIT :lim + """ + ).bindparams( + bindparam("doc_ids_document", expanding=True), + bindparam("doc_ids_doc", expanding=True), + bindparam("doc_ids_subject", expanding=True), + ) + if layers: + params["layers"] = [str(item).strip() for item in layers if str(item).strip()] + if params["layers"]: + stmt = text( + """ + SELECT path, content, layer, title, metadata_json, span_start, span_end, + 0 AS lexical_rank, + 0 AS prefer_bonus, + 0 AS test_penalty, + 0 AS structural_rank, + 0 AS layer_rank, + 0 AS distance + FROM rag_chunks + WHERE rag_session_id = :sid + AND layer IN :layers + AND ( + COALESCE(metadata_json->>'document_id', '') IN :doc_ids_document + OR COALESCE(metadata_json->>'doc_id', '') IN :doc_ids_doc + OR COALESCE(metadata_json->>'subject_id', '') IN :doc_ids_subject + ) + ORDER BY + CASE layer + WHEN 'D1_DOCUMENT_CATALOG' THEN 0 + WHEN 'D2_FACT_INDEX' THEN 1 + WHEN 'D0_DOC_CHUNKS' THEN 2 + WHEN 'D4_WORKFLOW_INDEX' THEN 3 + WHEN 'D6_INTEGRATION_INDEX' THEN 4 + WHEN 'D5_RELATION_GRAPH' THEN 5 + ELSE 9 + END, + path ASC, + COALESCE(span_start, 0) ASC, + COALESCE(chunk_index, 0) ASC + LIMIT :lim + """ + ).bindparams( + bindparam("layers", expanding=True), + bindparam("doc_ids_document", expanding=True), + bindparam("doc_ids_doc", expanding=True), + bindparam("doc_ids_subject", expanding=True), + ) + with get_engine().connect() as conn: + rows = conn.execute(stmt, params).mappings().fetchall() + return [self._row_to_dict(row) for row in rows] + def _row_to_dict(self, row) -> dict: data = dict(row) raw_metadata = data.pop("metadata_json") diff --git a/src/app/modules/rag/persistence/repository.py b/src/app/core/rag/persistence/repository.py similarity index 82% rename from src/app/modules/rag/persistence/repository.py rename to src/app/core/rag/persistence/repository.py index 43be993..cb99adb 100644 --- a/src/app/modules/rag/persistence/repository.py +++ b/src/app/core/rag/persistence/repository.py @@ -1,13 +1,13 @@ from __future__ import annotations -from app.modules.rag.contracts import RagDocument -from app.modules.rag.persistence.cache_repository import RagCacheRepository -from app.modules.rag.persistence.document_repository import RagDocumentRepository -from app.modules.rag.persistence.job_repository import RagJobRepository, RagJobRow -from app.modules.rag.persistence.query_repository import RagQueryRepository -from app.modules.rag.persistence.schema_repository import RagSchemaRepository -from app.modules.rag.persistence.session_repository import RagSessionRepository -from app.modules.shared.db import get_engine +from app.core.rag.contracts import RagDocument +from app.core.rag.persistence.cache_repository import RagCacheRepository +from app.core.rag.persistence.document_repository import RagDocumentRepository +from app.core.rag.persistence.job_repository import RagJobRepository, RagJobRow +from app.core.rag.persistence.query_repository import RagQueryRepository +from app.core.rag.persistence.schema_repository import RagSchemaRepository +from app.core.rag.persistence.session_repository import RagSessionRepository +from app.core.shared.database import get_engine class RagRepository: @@ -56,9 +56,6 @@ class RagRepository: def cache_documents(self, repo_id: str, path: str, blob_sha: str, docs: list[RagDocument]) -> None: self._cache.cache_documents(repo_id, path, blob_sha, docs) - def record_repo_cache(self, **kwargs) -> None: - self._cache.record_repo_cache(**kwargs) - def retrieve( self, rag_session_id: str, @@ -129,3 +126,18 @@ class RagRepository: layers=layers, limit=limit, ) + + def retrieve_by_document_ids( + self, + rag_session_id: str, + *, + document_ids: list[str], + layers: list[str] | None = None, + limit: int = 200, + ) -> list[dict]: + return self._query.retrieve_by_document_ids( + rag_session_id, + document_ids=document_ids, + layers=layers, + limit=limit, + ) diff --git a/src/app/modules/rag/persistence/retrieval_statement_builder.py b/src/app/core/rag/persistence/retrieval_statement_builder.py similarity index 97% rename from src/app/modules/rag/persistence/retrieval_statement_builder.py rename to src/app/core/rag/persistence/retrieval_statement_builder.py index 714c582..3fb34c3 100644 --- a/src/app/modules/rag/persistence/retrieval_statement_builder.py +++ b/src/app/core/rag/persistence/retrieval_statement_builder.py @@ -1,6 +1,6 @@ from __future__ import annotations -from app.modules.rag.retrieval.query_terms import extract_query_terms +from app.core.rag.retrieval.query_terms import extract_query_terms _LIKE_ESCAPE_SQL = " ESCAPE E'\\\\'" @@ -57,9 +57,10 @@ class RetrievalStatementBuilder: "WHEN layer = 'D1_DOCUMENT_CATALOG' THEN 0 " "WHEN layer = 'D4_WORKFLOW_INDEX' THEN 1 " "WHEN layer = 'D2_FACT_INDEX' THEN 2 " - "WHEN layer = 'D5_RELATION_GRAPH' THEN 3 " - "WHEN layer = 'D3_ENTITY_CATALOG' THEN 4 " - "WHEN layer = 'D0_DOC_CHUNKS' THEN 5 " + "WHEN layer = 'D6_INTEGRATION_INDEX' THEN 3 " + "WHEN layer = 'D5_RELATION_GRAPH' THEN 4 " + "WHEN layer = 'D3_ENTITY_CATALOG' THEN 5 " + "WHEN layer = 'D0_DOC_CHUNKS' THEN 6 " "ELSE 10 END" ) sql = f""" diff --git a/src/app/modules/rag/persistence/schema_migrator.py b/src/app/core/rag/persistence/schema_migrator.py similarity index 98% rename from src/app/modules/rag/persistence/schema_migrator.py rename to src/app/core/rag/persistence/schema_migrator.py index f552760..87e639e 100644 --- a/src/app/modules/rag/persistence/schema_migrator.py +++ b/src/app/core/rag/persistence/schema_migrator.py @@ -2,7 +2,7 @@ from __future__ import annotations from sqlalchemy import text -from app.modules.rag.persistence.schema_statements import legacy_chunk_columns +from app.core.rag.persistence.schema_statements import legacy_chunk_columns class RagSchemaMigrator: diff --git a/src/app/modules/rag/persistence/schema_repository.py b/src/app/core/rag/persistence/schema_repository.py similarity index 80% rename from src/app/modules/rag/persistence/schema_repository.py rename to src/app/core/rag/persistence/schema_repository.py index bab85c6..14ae742 100644 --- a/src/app/modules/rag/persistence/schema_repository.py +++ b/src/app/core/rag/persistence/schema_repository.py @@ -2,13 +2,13 @@ from __future__ import annotations from sqlalchemy import text -from app.modules.rag.persistence.schema_migrator import RagSchemaMigrator -from app.modules.rag.persistence.schema_statements import ( +from app.core.rag.persistence.schema_migrator import RagSchemaMigrator +from app.core.rag.persistence.schema_statements import ( base_table_statements, index_statements, legacy_drop_statements, ) -from app.modules.shared.db import get_engine +from app.core.shared.database import get_engine class RagSchemaRepository: diff --git a/src/app/modules/rag/persistence/schema_statements.py b/src/app/core/rag/persistence/schema_statements.py similarity index 100% rename from src/app/modules/rag/persistence/schema_statements.py rename to src/app/core/rag/persistence/schema_statements.py diff --git a/src/app/modules/rag/persistence/session_repository.py b/src/app/core/rag/persistence/session_repository.py similarity index 96% rename from src/app/modules/rag/persistence/session_repository.py rename to src/app/core/rag/persistence/session_repository.py index 5c36a5d..1b42289 100644 --- a/src/app/modules/rag/persistence/session_repository.py +++ b/src/app/core/rag/persistence/session_repository.py @@ -2,7 +2,7 @@ from __future__ import annotations from sqlalchemy import text -from app.modules.shared.db import get_engine +from app.core.shared.database import get_engine class RagSessionRepository: diff --git a/src/app/modules/rag/session_store.py b/src/app/core/rag/persistence/session_store.py similarity index 94% rename from src/app/modules/rag/session_store.py rename to src/app/core/rag/persistence/session_store.py index bc6de8c..527da6b 100644 --- a/src/app/modules/rag/session_store.py +++ b/src/app/core/rag/persistence/session_store.py @@ -3,7 +3,7 @@ from dataclasses import dataclass from uuid import uuid4 -from app.modules.rag.persistence.repository import RagRepository +from app.core.rag.persistence.repository import RagRepository @dataclass diff --git a/src/app/core/rag/retrieval/__init__.py b/src/app/core/rag/retrieval/__init__.py new file mode 100644 index 0000000..98ce1bc --- /dev/null +++ b/src/app/core/rag/retrieval/__init__.py @@ -0,0 +1,21 @@ +__all__ = ["TextChunker", "extract_query_terms", "RetrievalPlan", "RagSessionRetriever"] + + +def __getattr__(name: str): + if name == "TextChunker": + from app.core.rag.retrieval.chunker import TextChunker + + return TextChunker + if name == "extract_query_terms": + from app.core.rag.retrieval.query_terms import extract_query_terms + + return extract_query_terms + if name == "RetrievalPlan": + from app.core.rag.retrieval.session_retriever import RetrievalPlan + + return RetrievalPlan + if name == "RagSessionRetriever": + from app.core.rag.retrieval.session_retriever import RagSessionRetriever + + return RagSessionRetriever + raise AttributeError(name) diff --git a/src/app/modules/rag/retrieval/chunker.py b/src/app/core/rag/retrieval/chunker.py similarity index 100% rename from src/app/modules/rag/retrieval/chunker.py rename to src/app/core/rag/retrieval/chunker.py diff --git a/src/app/modules/rag/retrieval/query_terms.py b/src/app/core/rag/retrieval/query_terms.py similarity index 100% rename from src/app/modules/rag/retrieval/query_terms.py rename to src/app/core/rag/retrieval/query_terms.py diff --git a/src/app/modules/rag/retrieval/scoring.py b/src/app/core/rag/retrieval/scoring.py similarity index 100% rename from src/app/modules/rag/retrieval/scoring.py rename to src/app/core/rag/retrieval/scoring.py diff --git a/src/app/core/rag/retrieval/session_retriever.py b/src/app/core/rag/retrieval/session_retriever.py new file mode 100644 index 0000000..ea9004a --- /dev/null +++ b/src/app/core/rag/retrieval/session_retriever.py @@ -0,0 +1,97 @@ +"""Стандартный API retrieval по RAG-сессии: эмбеддинг запроса и поиск по плану слоёв.""" + +from __future__ import annotations + +import asyncio +from dataclasses import dataclass, field +from typing import TYPE_CHECKING + +from app.core.rag.embedding.gigachat_embedder import GigaChatEmbedder + +if TYPE_CHECKING: + from app.core.rag.persistence.repository import RagRepository + + +@dataclass(slots=True) +class RetrievalPlan: + """Параметры одного запроса к хранилищу: профиль, список слоёв и лимит строк.""" + + profile: str + layers: list[str] + limit: int + filters: dict[str, object] = field(default_factory=dict) + + +class RagSessionRetriever: + """Выполняет retrieve через :class:`RagRepository` и эмбеддер; единая точка поиска по сессии.""" + + def __init__(self, repository: RagRepository, embedder: GigaChatEmbedder) -> None: + self._repository = repository + self._embedder = embedder + + async def retrieve(self, rag_session_id: str, query_text: str, plan: RetrievalPlan) -> list[dict]: + """Возвращает сырые строки чанков (как в RagRepository.retrieve), с полем layer.""" + if not plan.layers: + return [] + query_embedding = await asyncio.to_thread(self._embed_query, query_text) + kwargs = self._map_filters(plan.filters) + return await asyncio.to_thread( + self._repository.retrieve, + rag_session_id, + query_embedding, + query_text=query_text, + layers=plan.layers, + limit=plan.limit, + **kwargs, + ) + + async def retrieve_exact_files( + self, + rag_session_id: str, + *, + paths: list[str], + layers: list[str] | None = None, + limit: int = 200, + ) -> list[dict]: + return await asyncio.to_thread( + self._repository.retrieve_exact_files, + rag_session_id, + paths=paths, + layers=layers, + limit=limit, + ) + + async def retrieve_chunks_by_path_substrings( + self, + rag_session_id: str, + *, + path_needles: list[str], + layers: list[str] | None = None, + limit: int = 200, + ) -> list[dict]: + return await asyncio.to_thread( + self._repository.retrieve_chunks_by_path_substrings, + rag_session_id, + path_needles=path_needles, + layers=layers, + limit=limit, + ) + + def _embed_query(self, query: str) -> list[float]: + return self._embedder.embed([query])[0] + + def _map_filters(self, filters: dict[str, object]) -> dict: + """Маппинг опциональных фильтров плана на kwargs RagRepository.retrieve.""" + out: dict = {} + keys = ( + "path_prefixes", + "exclude_path_prefixes", + "exclude_like_patterns", + "prefer_path_prefixes", + "prefer_like_patterns", + "prefer_non_tests", + ) + for key in keys: + if key in filters: + out[key] = filters[key] + return out diff --git a/src/app/modules/rag/retrieval/test_filter.py b/src/app/core/rag/retrieval/test_filter.py similarity index 100% rename from src/app/modules/rag/retrieval/test_filter.py rename to src/app/core/rag/retrieval/test_filter.py diff --git a/src/app/modules/shared/README.md b/src/app/core/shared/README.md similarity index 74% rename from src/app/modules/shared/README.md rename to src/app/core/shared/README.md index 0f49eea..b27a64a 100644 --- a/src/app/modules/shared/README.md +++ b/src/app/core/shared/README.md @@ -2,11 +2,12 @@ ## 1. Функции модуля - Общие инфраструктурные компоненты для всех модулей: - - подключение к БД (`db.py`, `bootstrap.py`), - - шина событий (`event_bus.py`), - - retry и idempotency (`retry_executor.py`, `idempotency_store.py`), - - checkpointer, - - клиент и настройки GigaChat (`gigachat/*`). + - `config/*` - загрузка окружения и общих настроек workspace, + - `database/*` - подключение к БД, bootstrap и checkpointer, + - `messaging/*` - общая шина событий и SSE-совместимые события, + - `resilience/*` - retry и idempotency, + - `observability/*` - runtime trace и trace logging, + - `gigachat/*` - клиент и настройки GigaChat. ## 2. Диаграмма классов и взаимосвязей ```mermaid @@ -25,10 +26,13 @@ classDiagram ## 3. Описание классов - `EventBus`: общий асинхронный event-bus для публикации и SSE-стриминга. Методы: `subscribe` — подписка на канал; `unsubscribe` — удаление подписки; `publish` — отправка события; `as_sse` — сериализация события в SSE-формат. +- `database_url/get_engine/get_session_factory`: общий доступ к engine и session factory SQLAlchemy. +- `bootstrap_database/get_checkpointer`: инициализация persistence-слоя и LangGraph checkpointer. - `RetryExecutor`: общий механизм повторных попыток для временных ошибок. Методы: `run` — выполняет async-операцию с retry по временным исключениям. - `IdempotencyStore`: in-memory хранилище идемпотентных ключей. Методы: `get_task_id` — находит ранее созданную задачу по ключу; `put` — сохраняет ключ и task_id. +- `load_workspace_env`: загрузка `.env` и `.env.local` по цепочке родительских директорий. - `GigaChatSettings`: конфигурация доступа к GigaChat. Методы: `from_env` — собирает настройки из переменных окружения. - `GigaChatTokenProvider`: управление access token для GigaChat API. diff --git a/src/app/modules/__init__.py b/src/app/core/shared/__init__.py similarity index 100% rename from src/app/modules/__init__.py rename to src/app/core/shared/__init__.py diff --git a/src/app/core/shared/config/__init__.py b/src/app/core/shared/config/__init__.py new file mode 100644 index 0000000..fc8fef7 --- /dev/null +++ b/src/app/core/shared/config/__init__.py @@ -0,0 +1,3 @@ +from app.core.shared.config.env_loader import load_workspace_env + +__all__ = ["load_workspace_env"] diff --git a/src/app/modules/shared/env_loader.py b/src/app/core/shared/config/env_loader.py similarity index 100% rename from src/app/modules/shared/env_loader.py rename to src/app/core/shared/config/env_loader.py diff --git a/src/app/core/shared/database/__init__.py b/src/app/core/shared/database/__init__.py new file mode 100644 index 0000000..60857f6 --- /dev/null +++ b/src/app/core/shared/database/__init__.py @@ -0,0 +1,13 @@ +from app.core.shared.database.bootstrap import bootstrap_database +from app.core.shared.database.checkpointer import get_checkpointer +from app.core.shared.database.db import database_url, get_engine, get_session_factory +from app.core.shared.database.readiness import DatabaseReadiness + +__all__ = [ + "DatabaseReadiness", + "bootstrap_database", + "database_url", + "get_checkpointer", + "get_engine", + "get_session_factory", +] diff --git a/src/app/modules/shared/bootstrap.py b/src/app/core/shared/database/bootstrap.py similarity index 90% rename from src/app/modules/shared/bootstrap.py rename to src/app/core/shared/database/bootstrap.py index ed90f3f..06d0693 100644 --- a/src/app/modules/shared/bootstrap.py +++ b/src/app/core/shared/database/bootstrap.py @@ -1,6 +1,6 @@ import time -from app.modules.shared.checkpointer import get_checkpointer +from app.core.shared.database.checkpointer import get_checkpointer def bootstrap_database(rag_repository, story_context_repository, chat_repository=None) -> None: diff --git a/src/app/modules/shared/checkpointer.py b/src/app/core/shared/database/checkpointer.py similarity index 81% rename from src/app/modules/shared/checkpointer.py rename to src/app/core/shared/database/checkpointer.py index 955f9bd..b4f59cc 100644 --- a/src/app/modules/shared/checkpointer.py +++ b/src/app/core/shared/database/checkpointer.py @@ -7,7 +7,7 @@ from langgraph.checkpoint.postgres import PostgresSaver from psycopg.errors import DuplicateColumn from psycopg.rows import dict_row -from app.modules.shared.db import database_url +from app.core.shared.database.db import database_url _CHECKPOINTER: PostgresSaver | None = None _LOCK = threading.Lock() @@ -29,8 +29,6 @@ def get_checkpointer() -> PostgresSaver: try: cp.setup() except DuplicateColumn as exc: - # LangGraph migrations may be partially applied in an existing DB. - # Treat duplicate task_path as already initialized schema. if "task_path" not in str(exc): raise _CHECKPOINTER = cp diff --git a/src/app/modules/shared/db.py b/src/app/core/shared/database/db.py similarity index 100% rename from src/app/modules/shared/db.py rename to src/app/core/shared/database/db.py diff --git a/src/app/core/shared/database/readiness.py b/src/app/core/shared/database/readiness.py new file mode 100644 index 0000000..5516f7d --- /dev/null +++ b/src/app/core/shared/database/readiness.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +from dataclasses import dataclass + +from app.infra.exceptions import AppError +from app.schemas.common import ModuleName + + +@dataclass +class DatabaseReadiness: + _error: str | None = None + + def mark_ready(self) -> None: + self._error = None + + def mark_unavailable(self, exc: Exception) -> None: + self._error = str(exc).strip() or exc.__class__.__name__ + + def require_ready(self) -> None: + if self._error is None: + return + raise AppError( + "database_unavailable", + "Database is unavailable. Configure DATABASE_URL or start the database service. " + f"Last error: {self._error}", + ModuleName.BACKEND, + ) + + def health_payload(self) -> dict[str, str]: + if self._error is None: + return {"status": "ok"} + return { + "status": "degraded", + "reason": "database_unavailable", + "details": self._error, + } + + @property + def error(self) -> str | None: + return self._error diff --git a/src/app/modules/agent/__init__.py b/src/app/core/shared/gigachat/__init__.py similarity index 100% rename from src/app/modules/agent/__init__.py rename to src/app/core/shared/gigachat/__init__.py diff --git a/src/app/modules/shared/gigachat/client.py b/src/app/core/shared/gigachat/client.py similarity index 92% rename from src/app/modules/shared/gigachat/client.py rename to src/app/core/shared/gigachat/client.py index 5096bbe..0127295 100644 --- a/src/app/modules/shared/gigachat/client.py +++ b/src/app/core/shared/gigachat/client.py @@ -2,10 +2,10 @@ import time import requests -from app.core.constants import MAX_RETRIES -from app.modules.shared.gigachat.errors import GigaChatError -from app.modules.shared.gigachat.settings import GigaChatSettings -from app.modules.shared.gigachat.token_provider import GigaChatTokenProvider +from app.infra.constants import MAX_RETRIES +from app.core.shared.gigachat.errors import GigaChatError +from app.core.shared.gigachat.settings import GigaChatSettings +from app.core.shared.gigachat.token_provider import GigaChatTokenProvider class GigaChatClient: diff --git a/src/app/modules/shared/gigachat/errors.py b/src/app/core/shared/gigachat/errors.py similarity index 100% rename from src/app/modules/shared/gigachat/errors.py rename to src/app/core/shared/gigachat/errors.py diff --git a/src/app/modules/shared/gigachat/settings.py b/src/app/core/shared/gigachat/settings.py similarity index 100% rename from src/app/modules/shared/gigachat/settings.py rename to src/app/core/shared/gigachat/settings.py diff --git a/src/app/modules/shared/gigachat/token_provider.py b/src/app/core/shared/gigachat/token_provider.py similarity index 93% rename from src/app/modules/shared/gigachat/token_provider.py rename to src/app/core/shared/gigachat/token_provider.py index 7fb790f..0d8adb6 100644 --- a/src/app/modules/shared/gigachat/token_provider.py +++ b/src/app/core/shared/gigachat/token_provider.py @@ -4,8 +4,8 @@ import uuid import requests -from app.modules.shared.gigachat.errors import GigaChatError -from app.modules.shared.gigachat.settings import GigaChatSettings +from app.core.shared.gigachat.errors import GigaChatError +from app.core.shared.gigachat.settings import GigaChatSettings class GigaChatTokenProvider: diff --git a/src/app/core/shared/messaging/__init__.py b/src/app/core/shared/messaging/__init__.py new file mode 100644 index 0000000..f3261da --- /dev/null +++ b/src/app/core/shared/messaging/__init__.py @@ -0,0 +1,3 @@ +from app.core.shared.messaging.event_bus import Event, EventBus + +__all__ = ["Event", "EventBus"] diff --git a/src/app/modules/shared/event_bus.py b/src/app/core/shared/messaging/event_bus.py similarity index 100% rename from src/app/modules/shared/event_bus.py rename to src/app/core/shared/messaging/event_bus.py diff --git a/src/app/core/shared/resilience/__init__.py b/src/app/core/shared/resilience/__init__.py new file mode 100644 index 0000000..14439aa --- /dev/null +++ b/src/app/core/shared/resilience/__init__.py @@ -0,0 +1,4 @@ +from app.core.shared.resilience.idempotency_store import IdempotencyRecord, IdempotencyStore +from app.core.shared.resilience.retry_executor import RetryExecutor + +__all__ = ["IdempotencyRecord", "IdempotencyStore", "RetryExecutor"] diff --git a/src/app/modules/shared/idempotency_store.py b/src/app/core/shared/resilience/idempotency_store.py similarity index 82% rename from src/app/modules/shared/idempotency_store.py rename to src/app/core/shared/resilience/idempotency_store.py index 5434988..000b269 100644 --- a/src/app/modules/shared/idempotency_store.py +++ b/src/app/core/shared/resilience/idempotency_store.py @@ -2,7 +2,7 @@ from dataclasses import dataclass from datetime import datetime, timezone from threading import Lock -from app.core.constants import IDEMPOTENCY_TTL +from app.infra.constants import IDEMPOTENCY_TTL @dataclass @@ -31,10 +31,6 @@ class IdempotencyStore: ) def _cleanup_locked(self, now: datetime) -> None: - expired = [ - key - for key, rec in self._records.items() - if now - rec.created_at > IDEMPOTENCY_TTL - ] + expired = [key for key, rec in self._records.items() if now - rec.created_at > IDEMPOTENCY_TTL] for key in expired: del self._records[key] diff --git a/src/app/modules/shared/retry_executor.py b/src/app/core/shared/resilience/retry_executor.py similarity index 93% rename from src/app/modules/shared/retry_executor.py rename to src/app/core/shared/resilience/retry_executor.py index eb0e89b..7b70f0a 100644 --- a/src/app/modules/shared/retry_executor.py +++ b/src/app/core/shared/resilience/retry_executor.py @@ -1,7 +1,7 @@ import asyncio from typing import Awaitable, Callable, TypeVar -from app.core.constants import MAX_RETRIES +from app.infra.constants import MAX_RETRIES T = TypeVar("T") diff --git a/src/app/core/shared/story_context/__init__.py b/src/app/core/shared/story_context/__init__.py new file mode 100644 index 0000000..76e0023 --- /dev/null +++ b/src/app/core/shared/story_context/__init__.py @@ -0,0 +1,3 @@ +from app.core.shared.story_context.schema_repository import StoryContextSchemaRepository + +__all__ = ["StoryContextSchemaRepository"] diff --git a/src/app/core/shared/story_context/schema_repository.py b/src/app/core/shared/story_context/schema_repository.py new file mode 100644 index 0000000..3ffd53c --- /dev/null +++ b/src/app/core/shared/story_context/schema_repository.py @@ -0,0 +1,126 @@ +from __future__ import annotations + +from sqlalchemy import text + +from app.core.shared.database import get_engine + + +class StoryContextSchemaRepository: + def ensure_tables(self) -> None: + with get_engine().connect() as conn: + for statement in self._statements(): + conn.execute(text(statement)) + conn.commit() + + def _statements(self) -> tuple[str, ...]: + return ( + """ + CREATE TABLE IF NOT EXISTS story_records ( + story_id VARCHAR(128) PRIMARY KEY, + project_id VARCHAR(512) NOT NULL, + title TEXT NOT NULL, + status VARCHAR(64) NOT NULL DEFAULT 'draft', + owner VARCHAR(256) NULL, + metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb, + baseline_commit_sha VARCHAR(128) NULL, + snapshot_id VARCHAR(128) NULL, + created_by VARCHAR(256) NULL, + updated_by VARCHAR(256) NULL, + created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP + ) + """, + """ + CREATE TABLE IF NOT EXISTS story_artifacts ( + id BIGSERIAL PRIMARY KEY, + story_id VARCHAR(128) NOT NULL, + artifact_type VARCHAR(64) NOT NULL, + revision INTEGER NOT NULL, + content TEXT NOT NULL, + content_hash VARCHAR(128) NULL, + rag_session_id VARCHAR(64) NULL, + created_by VARCHAR(256) NULL, + created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, + artifact_role VARCHAR(64) NULL, + doc_id TEXT NULL, + doc_version TEXT NULL, + path TEXT NULL, + section TEXT NULL, + chunk_id TEXT NULL, + change_type VARCHAR(32) NULL, + summary TEXT NULL, + source_ref TEXT NULL, + session_id VARCHAR(128) NULL, + CONSTRAINT fk_story_artifacts_story + FOREIGN KEY (story_id) + REFERENCES story_records(story_id) + ON DELETE CASCADE, + CONSTRAINT uq_story_artifact_revision + UNIQUE (story_id, artifact_type, revision) + ) + """, + """ + CREATE TABLE IF NOT EXISTS story_links ( + id BIGSERIAL PRIMARY KEY, + story_id VARCHAR(128) NOT NULL, + link_type VARCHAR(64) NOT NULL, + link_value TEXT NOT NULL, + target_ref TEXT NULL, + description TEXT NULL, + metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb, + created_by VARCHAR(256) NULL, + created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT fk_story_links_story + FOREIGN KEY (story_id) + REFERENCES story_records(story_id) + ON DELETE CASCADE + ) + """, + """ + CREATE TABLE IF NOT EXISTS session_artifacts ( + id BIGSERIAL PRIMARY KEY, + session_id VARCHAR(128) NOT NULL, + project_id VARCHAR(512) NOT NULL, + artifact_role VARCHAR(64) NOT NULL, + source_ref TEXT NULL, + doc_id TEXT NULL, + doc_version TEXT NULL, + path TEXT NULL, + section TEXT NULL, + chunk_id TEXT NULL, + change_type VARCHAR(32) NULL, + summary TEXT NULL, + created_by VARCHAR(256) NULL, + created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, + bound_story_id VARCHAR(128) NULL, + bound_at TIMESTAMPTZ NULL + ) + """, + "ALTER TABLE story_records ADD COLUMN IF NOT EXISTS owner VARCHAR(256) NULL", + "ALTER TABLE story_records ADD COLUMN IF NOT EXISTS metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb", + "ALTER TABLE story_records ADD COLUMN IF NOT EXISTS baseline_commit_sha VARCHAR(128) NULL", + "ALTER TABLE story_records ADD COLUMN IF NOT EXISTS snapshot_id VARCHAR(128) NULL", + "ALTER TABLE story_records ADD COLUMN IF NOT EXISTS created_by VARCHAR(256) NULL", + "ALTER TABLE story_records ADD COLUMN IF NOT EXISTS updated_by VARCHAR(256) NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS artifact_role VARCHAR(64) NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS doc_id TEXT NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS doc_version TEXT NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS path TEXT NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS section TEXT NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS chunk_id TEXT NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS change_type VARCHAR(32) NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS summary TEXT NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS source_ref TEXT NULL", + "ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS session_id VARCHAR(128) NULL", + "ALTER TABLE story_links ADD COLUMN IF NOT EXISTS target_ref TEXT NULL", + "ALTER TABLE story_links ADD COLUMN IF NOT EXISTS description TEXT NULL", + "ALTER TABLE story_links ADD COLUMN IF NOT EXISTS metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb", + "ALTER TABLE story_links ADD COLUMN IF NOT EXISTS created_by VARCHAR(256) NULL", + "ALTER TABLE session_artifacts ADD COLUMN IF NOT EXISTS bound_story_id VARCHAR(128) NULL", + "ALTER TABLE session_artifacts ADD COLUMN IF NOT EXISTS bound_at TIMESTAMPTZ NULL", + "CREATE INDEX IF NOT EXISTS idx_story_records_project ON story_records(project_id, updated_at DESC)", + "CREATE INDEX IF NOT EXISTS idx_story_artifacts_story_type ON story_artifacts(story_id, artifact_type, revision DESC)", + "CREATE INDEX IF NOT EXISTS idx_story_links_story ON story_links(story_id, link_type)", + "CREATE INDEX IF NOT EXISTS idx_session_artifacts_session ON session_artifacts(session_id, created_at DESC)", + "CREATE INDEX IF NOT EXISTS idx_session_artifacts_project ON session_artifacts(project_id, created_at DESC)", + ) diff --git a/src/app/modules/shared/__init__.py b/src/app/infra/__init__.py similarity index 100% rename from src/app/modules/shared/__init__.py rename to src/app/infra/__init__.py diff --git a/src/app/core/constants.py b/src/app/infra/constants.py similarity index 100% rename from src/app/core/constants.py rename to src/app/infra/constants.py diff --git a/src/app/core/error_handlers.py b/src/app/infra/error_handlers.py similarity index 85% rename from src/app/core/error_handlers.py rename to src/app/infra/error_handlers.py index 52e8ff3..0909155 100644 --- a/src/app/core/error_handlers.py +++ b/src/app/infra/error_handlers.py @@ -2,7 +2,7 @@ from fastapi import FastAPI, Request from fastapi.responses import JSONResponse from pydantic import ValidationError -from app.core.exceptions import AppError +from app.infra.exceptions import AppError from app.schemas.common import ModuleName @@ -10,7 +10,7 @@ def register_error_handlers(app: FastAPI) -> None: @app.exception_handler(AppError) async def app_error_handler(_: Request, exc: AppError) -> JSONResponse: return JSONResponse( - status_code=400, + status_code=_status_code(exc), content={"code": exc.code, "desc": exc.desc, "module": exc.module.value}, ) @@ -35,3 +35,9 @@ def register_error_handlers(app: FastAPI) -> None: "module": ModuleName.BACKEND.value, }, ) + + +def _status_code(exc: AppError) -> int: + if exc.code == "database_unavailable": + return 503 + return 400 diff --git a/src/app/core/exceptions.py b/src/app/infra/exceptions.py similarity index 100% rename from src/app/core/exceptions.py rename to src/app/infra/exceptions.py diff --git a/src/app/core/logging_setup.py b/src/app/infra/logging_setup.py similarity index 100% rename from src/app/core/logging_setup.py rename to src/app/infra/logging_setup.py diff --git a/src/app/infra/observability/__init__.py b/src/app/infra/observability/__init__.py new file mode 100644 index 0000000..abb9632 --- /dev/null +++ b/src/app/infra/observability/__init__.py @@ -0,0 +1,13 @@ +from app.infra.observability.module_trace import ModuleTrace, ModuleTraceLogger, RequestTraceContext +from app.infra.observability.request_trace_logger import RequestTraceLogger +from app.infra.observability.trace_file_path_builder import TraceFilePathBuilder +from app.infra.observability.trace_markdown_writer import TraceMarkdownWriter + +__all__ = [ + "ModuleTrace", + "ModuleTraceLogger", + "RequestTraceContext", + "RequestTraceLogger", + "TraceFilePathBuilder", + "TraceMarkdownWriter", +] diff --git a/src/app/modules/agent/observability/module_trace.py b/src/app/infra/observability/module_trace.py similarity index 100% rename from src/app/modules/agent/observability/module_trace.py rename to src/app/infra/observability/module_trace.py diff --git a/src/app/modules/api/infrastructure/logging/request_trace_logger.py b/src/app/infra/observability/request_trace_logger.py similarity index 55% rename from src/app/modules/api/infrastructure/logging/request_trace_logger.py rename to src/app/infra/observability/request_trace_logger.py index 11001c9..7642950 100644 --- a/src/app/modules/api/infrastructure/logging/request_trace_logger.py +++ b/src/app/infra/observability/request_trace_logger.py @@ -3,11 +3,11 @@ from __future__ import annotations from pathlib import Path from threading import Lock -from app.modules.api.domain.events.client_event import ClientEventRecord -from app.modules.api.domain.models.agent_request import AgentRequest -from app.modules.api.domain.models.agent_session import AgentSession -from app.modules.api.infrastructure.logging.trace_file_path_builder import TraceFilePathBuilder -from app.modules.api.infrastructure.logging.trace_markdown_writer import TraceMarkdownWriter +from app.core.api.domain.events.client_event import ClientEventRecord +from app.core.api.domain.models.agent_request import AgentRequest +from app.core.api.domain.models.agent_session import AgentSession +from app.infra.observability.trace_file_path_builder import TraceFilePathBuilder +from app.infra.observability.trace_markdown_writer import TraceMarkdownWriter class RequestTraceLogger: @@ -15,23 +15,31 @@ class RequestTraceLogger: self._paths = TraceFilePathBuilder(root) self._writer = TraceMarkdownWriter() self._files: dict[str, Path] = {} + self._rag_session_files: dict[str, Path] = {} self._lock = Lock() def start_request(self, request: AgentRequest, session: AgentSession) -> None: - path = self._paths.build(request.request_id) - self._writer.initialize( + path = self._resolve_path(session.active_rag_session_id) + if not path.exists(): + self._writer.initialize( + path, + [ + f"# Runtime Trace: {path.stem}", + "", + f"- active_rag_session_id: {session.active_rag_session_id or '—'}", + ], + ) + self._writer.append_section( path, - [ - f"# Request Trace: {request.request_id}", - "", - f"- session_id: {session.session_id}", - f"- active_rag_session_id: {session.active_rag_session_id or '—'}", - f"- process_version: {request.process_version}", - f"- created_at: {request.created_at.isoformat()}", - "", - "## User Message", - request.message, - ], + "request", + { + "request_id": request.request_id, + "session_id": session.session_id, + "active_rag_session_id": session.active_rag_session_id, + "process_version": request.process_version, + "created_at": request.created_at.isoformat(), + "message": request.message, + }, ) with self._lock: self._files[request.request_id] = path @@ -45,17 +53,7 @@ class RequestTraceLogger: self._append(request_id, module, body) def log_event(self, event: ClientEventRecord) -> None: - self._append( - event.request_id, - "client_event", - { - "event": event.type.value, - "source": event.source, - "text": event.text, - "payload": event.payload, - "created_at": event.created_at.isoformat(), - }, - ) + return def complete_request(self, request: AgentRequest) -> None: self._append( @@ -85,3 +83,13 @@ class RequestTraceLogger: if path is None: return self._writer.append_section(path, title, payload) + + def _resolve_path(self, rag_session_id: str | None) -> Path: + if not rag_session_id: + return self._paths.build_for_rag_session(None) + with self._lock: + path = self._rag_session_files.get(rag_session_id) + if path is None: + path = self._paths.build_for_rag_session(rag_session_id) + self._rag_session_files[rag_session_id] = path + return path diff --git a/src/app/infra/observability/trace_file_path_builder.py b/src/app/infra/observability/trace_file_path_builder.py new file mode 100644 index 0000000..6dafacd --- /dev/null +++ b/src/app/infra/observability/trace_file_path_builder.py @@ -0,0 +1,21 @@ +from __future__ import annotations + +from datetime import datetime, timezone +from pathlib import Path + + +class TraceFilePathBuilder: + def __init__(self, root: Path) -> None: + self._root = root + + def build_for_rag_session(self, rag_session_id: str | None) -> Path: + self._root.mkdir(parents=True, exist_ok=True) + stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S") + suffix = self._suffix_from_rag_session(rag_session_id) + return self._root / f"{stamp}-{suffix}.md" + + def _suffix_from_rag_session(self, rag_session_id: str | None) -> str: + normalized = (rag_session_id or "").strip() + if not normalized: + return "no-rag-session" + return normalized[-12:] diff --git a/src/app/modules/api/infrastructure/logging/trace_markdown_writer.py b/src/app/infra/observability/trace_markdown_writer.py similarity index 100% rename from src/app/modules/api/infrastructure/logging/trace_markdown_writer.py rename to src/app/infra/observability/trace_markdown_writer.py diff --git a/src/app/main.py b/src/app/main.py index 30819d9..3dcb41f 100644 --- a/src/app/main.py +++ b/src/app/main.py @@ -3,9 +3,9 @@ import logging from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware -from app.core.logging_setup import configure_logging -from app.core.error_handlers import register_error_handlers -from app.modules.application import ModularApplication +from app.infra.logging_setup import configure_logging +from app.infra.error_handlers import register_error_handlers +from app.core.application import ModularApplication def _configure_logging() -> None: @@ -28,10 +28,6 @@ def create_app() -> FastAPI: ) app.include_router(modules.api.public_router()) - app.include_router(modules.rag.public_router()) - app.include_router(modules.rag.internal_router()) - app.include_router(modules.rag_repo.internal_router()) - register_error_handlers(app) @app.on_event("startup") @@ -40,7 +36,7 @@ def create_app() -> FastAPI: @app.get("/health") async def health() -> dict: - return {"status": "ok"} + return modules.health_payload() return app diff --git a/src/app/modules/agent/README.md b/src/app/modules/agent/README.md deleted file mode 100644 index 92d3ed5..0000000 --- a/src/app/modules/agent/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# Модуль agent - -## 1. Назначение -Модуль обеспечивает выполнение code-QA пайплайна для pipeline_setup_v3 и интеграцию с chat-слоем через адаптер к контракту `AgentRunner`. Оркестрация основана на **IntentRouterV2** (RAG) и **AgentRuntimeExecutor** (роутинг → retrieval → evidence gate → генерация ответа). - -## 2. Состав модуля -- **runtime/** — единственный orchestration-слой. На верхнем уровне содержит только файлы рантайма, а шаги исполнения вынесены в `runtime/steps/*` (`retrieval`, `context`, `gates`, `answer_policy`, `generation`, `finalization`, `explain`). Публичный API: `AgentRuntimeExecutor`, `RuntimeRetrievalAdapter`, `RuntimeRepoContextFactory`, модели `Runtime*`. -- **llm/** — сервис вызова LLM (GigaChat) с загрузкой системных промптов через `PromptLoader`. -- **llm/prompt_loader.py** — загрузка системных промптов из `llm/prompts.yml`. -- **runtime/code_qa_runner_adapter.py** — адаптер `AgentRuntimeExecutor` к протоколу `AgentRunner` для использования из chat (async `run` → sync `execute` в executor). - -## 3. Диаграмма зависимостей -```mermaid -classDiagram - class AgentRuntimeExecutor - class CodeQaRunnerAdapter - class AgentLlmService - class PromptLoader - class IntentRouterV2 - class RuntimeRetrievalAdapter - - CodeQaRunnerAdapter --> AgentRuntimeExecutor - AgentRuntimeExecutor --> AgentLlmService - AgentRuntimeExecutor --> IntentRouterV2 - AgentRuntimeExecutor --> RuntimeRetrievalAdapter - AgentLlmService --> PromptLoader -``` - -## 4. Точки входа -- **Тесты pipeline_setup_v3**: `AgentRuntimeAdapter` импортирует `AgentRuntimeExecutor`, `IntentRouterV2`, `RuntimeRepoContextFactory`, `RuntimeRetrievalAdapter`, `AgentLlmService`, `PromptLoader` из `app.modules.agent.runtime` и соответствующих пакетов. -- **Приложение (chat)**: `ModularApplication` собирает `AgentRuntimeExecutor` и оборачивает его в `CodeQaRunnerAdapter`; chat передаёт адаптер как `agent_runner` в `ChatModule`. - -## 5. Промпты -Используются только промпты, загружаемые из `llm/prompts.yml`: -- **code_qa_*** — ответы по sub_intent (architecture, explain, find_entrypoints, find_tests, general, open_file, trace_flow, degraded, repair). -- **rag_intent_router_v2** — классификация интента в IntentRouterV2. -- **code_explain_answer_v2** — прямой code-explain в chat (direct_service). diff --git a/src/app/modules/agent/intent_router_v2/__init__.py b/src/app/modules/agent/intent_router_v2/__init__.py deleted file mode 100644 index 36d8d2e..0000000 --- a/src/app/modules/agent/intent_router_v2/__init__.py +++ /dev/null @@ -1,19 +0,0 @@ -from app.modules.agent.intent_router_v2.models import ( - ConversationState, - IntentDecision, - IntentRouterResult, - QueryAnchor, - QueryPlan, - RepoContext, -) -from app.modules.agent.intent_router_v2.router import IntentRouterV2 - -__all__ = [ - "ConversationState", - "IntentDecision", - "IntentRouterResult", - "IntentRouterV2", - "QueryAnchor", - "QueryPlan", - "RepoContext", -] diff --git a/src/app/modules/agent/intent_router_v2/analysis/__init__.py b/src/app/modules/agent/intent_router_v2/analysis/__init__.py deleted file mode 100644 index 7dd31b8..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -from app.modules.agent.intent_router_v2.analysis.normalization import QueryNormalizer -from app.modules.agent.intent_router_v2.analysis.query_plan_builder import QueryPlanBuilder - -__all__ = ["QueryNormalizer", "QueryPlanBuilder"] diff --git a/src/app/modules/agent/intent_router_v2/analysis/ambiguity_detector.py b/src/app/modules/agent/intent_router_v2/analysis/ambiguity_detector.py deleted file mode 100644 index ba5b974..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/ambiguity_detector.py +++ /dev/null @@ -1,64 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.analysis.docs_query_signals import DocsQuerySignals -from app.modules.agent.intent_router_v2.analysis.docs_sub_intent_detector import DocsSubIntentDetector - - -class DocsAmbiguityDetector: - _ALLOWED = { - "SYSTEM_FLOW_EXPLAIN", - "COMPONENT_EXPLAIN", - "API_METHOD_EXPLAIN", - "ENTITY_EXPLAIN", - "RELATED_DOCS_EXPLAIN", - "GENERIC_QA", - } - - def __init__( - self, - docs_signals: DocsQuerySignals | None = None, - detector: DocsSubIntentDetector | None = None, - ) -> None: - self._docs_signals = docs_signals or DocsQuerySignals() - self._detector = detector or DocsSubIntentDetector() - self._natural_entity_markers = ("runtime health", "статус воркера", "состояние runtime", "состояние воркера") - self._natural_flow_markers = ("health check runtime", "проверка состояния", "как работает health check", "как происходит проверка") - - def detect(self, query: str, *, intent: str, sub_intent: str) -> dict[str, object]: - if intent not in {"DOCUMENTATION_EXPLAIN", "GENERAL_QA"}: - return self._result(False, "", sub_intent, []) - candidates = [name for name, _ in self._detector.rank_candidates(query, intent="DOCUMENTATION_EXPLAIN") if name in self._ALLOWED] - if not candidates: - return self._result(False, "", sub_intent, []) - top_scores = self._detector.rank_candidates(query, intent="DOCUMENTATION_EXPLAIN") - score_map = {name: score for name, score in top_scores} - selected_score = int(score_map.get(sub_intent, 0)) - second_score = int(next((score for name, score in top_scores if name != sub_intent), 0)) - has_strong_anchor = bool(self._docs_signals.query_anchor_candidates(query)) or self._docs_signals.has_component_like_token(query) - if sub_intent == "GENERIC_QA" and has_strong_anchor: - return self._result(True, "general_vs_anchor_conflict", sub_intent, candidates[:4]) - if sub_intent != "GENERIC_QA" and self._looks_overview_like(query) and not has_strong_anchor: - return self._result(True, "overview_vs_explain_conflict", sub_intent, candidates[:4]) - if not has_strong_anchor and self._looks_natural_boundary(query): - return self._result(True, "natural_language_boundary", sub_intent, candidates[:4]) - if selected_score and second_score and selected_score - second_score <= 1 and second_score >= 2: - return self._result(True, "small_score_gap", sub_intent, candidates[:4]) - if not has_strong_anchor and selected_score <= 3 and second_score >= 1: - return self._result(True, "weak_deterministic_signal", sub_intent, candidates[:4]) - return self._result(False, "", sub_intent, candidates[:4]) - - def _looks_overview_like(self, query: str) -> bool: - text = " ".join((query or "").lower().split()) - return any(marker in text for marker in ("что есть в документации", "какая структура документации", "что описано", "обзор документации", "с чего начать")) - - def _looks_natural_boundary(self, query: str) -> bool: - text = " ".join((query or "").lower().split()) - return any(marker in text for marker in (*self._natural_entity_markers, *self._natural_flow_markers)) - - def _result(self, is_ambiguous: bool, reason: str, selected: str, candidates: list[str]) -> dict[str, object]: - return { - "is_ambiguous": is_ambiguous, - "ambiguity_reason": reason, - "deterministic_candidates": list(candidates), - "deterministic_primary_candidate": selected, - } diff --git a/src/app/modules/agent/intent_router_v2/analysis/anchor_extractor.py b/src/app/modules/agent/intent_router_v2/analysis/anchor_extractor.py deleted file mode 100644 index 3f1555c..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/anchor_extractor.py +++ /dev/null @@ -1,144 +0,0 @@ -from __future__ import annotations - -import re - -from app.modules.agent.intent_router_v2.models import AnchorSpan, QueryAnchor -from app.modules.agent.intent_router_v2.analysis.normalization_terms import KeyTermCanonicalizer -from app.modules.agent.intent_router_v2.analysis.symbol_rules import COMMON_PATH_SEGMENTS, PY_KEYWORDS -from app.modules.agent.intent_router_v2.analysis.term_mapping import RuEnTermMapper - -_FILE_PATTERN = re.compile(r"(?P\b(?:[\w.-]+/)*[\w.-]+\.(?:py|md|rst|txt|yaml|yml|json|toml|ini|cfg)\b)") -_PATH_HINT_PATTERN = re.compile(r"(?P\b(?:src|app|docs|tests)/[\w./-]*[\w-]\b)") -_SYMBOL_PATTERN = re.compile( - r"\b(?P[A-Z][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)+|[A-Z][A-Za-z0-9_]{2,}|[a-z_][A-Za-z0-9_]{2,})\b" -) -_DOC_SECTION_PATTERN = re.compile(r"(?:section|раздел)\s+[\"'`#]?(?P[A-Za-zА-Яа-я0-9_ ./:-]{2,})", re.IGNORECASE) -_WORD_RE = re.compile(r"[A-Za-zА-Яа-яЁё-]+") - - -class AnchorExtractor: - def __init__( - self, - mapper: RuEnTermMapper | None = None, - canonicalizer: KeyTermCanonicalizer | None = None, - ) -> None: - self._mapper = mapper or RuEnTermMapper() - self._canonicalizer = canonicalizer or KeyTermCanonicalizer() - - def extract(self, text: str) -> list[QueryAnchor]: - anchors = self._file_anchors(text) - anchors.extend(self._symbol_anchors(text, file_anchors=anchors)) - anchors.extend(self._doc_ref_anchors(text)) - anchors.extend(self._key_term_anchors(text)) - return self._dedupe(anchors) - - def _file_anchors(self, text: str) -> list[QueryAnchor]: - anchors = self._anchors_from_matches(_FILE_PATTERN.finditer(text), anchor_type="FILE_PATH", confidence=0.95) - anchors.extend(self._anchors_from_matches(_PATH_HINT_PATTERN.finditer(text), anchor_type="FILE_PATH", confidence=0.8)) - return anchors - - def _symbol_anchors(self, text: str, *, file_anchors: list[QueryAnchor]) -> list[QueryAnchor]: - anchors: list[QueryAnchor] = [] - path_ranges = [(anchor.span.start, anchor.span.end) for anchor in file_anchors if anchor.span is not None] - path_segments = self._path_segments(file_anchors) - for match in _SYMBOL_PATTERN.finditer(text): - value = match.group("value") - if value.endswith((".py", ".md")) or "/" in value: - continue - if self._is_inside_path(match.start("value"), match.end("value"), path_ranges): - continue - if self._is_keyword(value): - continue - if file_anchors and value.lower() in path_segments: - continue - anchors.append(self._anchor("SYMBOL", value, match.start("value"), match.end("value"), 0.88, source="user_text")) - return anchors - - def _doc_ref_anchors(self, text: str) -> list[QueryAnchor]: - anchors = self._anchors_from_matches(_DOC_SECTION_PATTERN.finditer(text), anchor_type="DOC_REF", confidence=0.75, subtype="section") - for match in _FILE_PATTERN.finditer(text): - value = match.group("value") - if not value.lower().endswith((".md", ".rst", ".txt")): - continue - anchors.append(self._anchor("DOC_REF", value, match.start("value"), match.end("value"), 0.92, subtype="file", source="user_text")) - return anchors - - def _key_term_anchors(self, text: str) -> list[QueryAnchor]: - literals = set(self._mapper.all_literal_terms()) - anchors: list[QueryAnchor] = [] - for token in _WORD_RE.finditer(text): - value = token.group(0) - normalized = value.lower() - canonical = self._canonicalizer.canonicalize(value) - if canonical is None and normalized not in literals: - continue - anchors.append( - self._anchor( - "KEY_TERM", - canonical or value, - token.start(), - token.end(), - 0.9, - source="user_text", - ) - ) - return anchors - - def _anchors_from_matches( - self, - matches, - *, - anchor_type: str, - confidence: float, - subtype: str | None = None, - ) -> list[QueryAnchor]: - return [ - self._anchor(anchor_type, match.group("value"), match.start("value"), match.end("value"), confidence, subtype=subtype) - for match in matches - ] - - def _anchor( - self, - anchor_type: str, - value: str, - start: int, - end: int, - confidence: float, - subtype: str | None = None, - source: str = "user_text", - ) -> QueryAnchor: - return QueryAnchor( - type=anchor_type, - value=value, - subtype=subtype, - source=source, - span=AnchorSpan(start=start, end=end), - confidence=confidence, - ) - - def _dedupe(self, anchors: list[QueryAnchor]) -> list[QueryAnchor]: - result: list[QueryAnchor] = [] - seen: set[tuple[str, str, str | None, str]] = set() - for anchor in anchors: - key = (anchor.type, anchor.value, anchor.subtype, anchor.source) - if key in seen: - continue - seen.add(key) - result.append(anchor) - return result - - def _is_inside_path(self, start: int, end: int, ranges: list[tuple[int, int]]) -> bool: - return any(start >= left and end <= right for left, right in ranges) - - def _is_keyword(self, token: str) -> bool: - return token.lower() in PY_KEYWORDS - - def _path_segments(self, anchors: list[QueryAnchor]) -> set[str]: - values: set[str] = set() - for anchor in anchors: - parts = re.split(r"[/.]+", anchor.value.lower()) - for part in parts: - if not part: - continue - values.add(part) - return values | COMMON_PATH_SEGMENTS diff --git a/src/app/modules/agent/intent_router_v2/analysis/anchor_span_validator.py b/src/app/modules/agent/intent_router_v2/analysis/anchor_span_validator.py deleted file mode 100644 index 1600a06..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/anchor_span_validator.py +++ /dev/null @@ -1,22 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.models import QueryAnchor - - -class AnchorSpanValidator: - def sanitize(self, anchors: list[QueryAnchor], raw_len: int) -> list[QueryAnchor]: - result: list[QueryAnchor] = [] - for anchor in anchors: - if anchor.source != "user_text": - result.append(anchor.model_copy(update={"span": None})) - continue - if anchor.span is None: - result.append(anchor) - continue - start = int(anchor.span.start) - end = int(anchor.span.end) - if 0 <= start < end <= raw_len: - result.append(anchor) - continue - result.append(anchor.model_copy(update={"span": None, "confidence": max(anchor.confidence * 0.5, 0.0)})) - return result diff --git a/src/app/modules/agent/intent_router_v2/analysis/conversation_anchor_builder.py b/src/app/modules/agent/intent_router_v2/analysis/conversation_anchor_builder.py deleted file mode 100644 index d258266..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/conversation_anchor_builder.py +++ /dev/null @@ -1,61 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.analysis.followup_detector import FollowUpDetector -from app.modules.agent.intent_router_v2.models import ConversationState, QueryAnchor - - -class ConversationAnchorBuilder: - def __init__(self, followup: FollowUpDetector | None = None) -> None: - self._followup = followup or FollowUpDetector() - - def build( - self, - raw: str, - state: ConversationState, - *, - continue_mode: bool, - has_user_symbol: bool, - has_user_file_path: bool, - ) -> list[QueryAnchor]: - if not continue_mode: - return [] - anchors: list[QueryAnchor] = [] - if has_user_file_path: - return anchors - for path in self._paths_for_carryover(state.active_path_scope): - anchors.append( - QueryAnchor( - type="FILE_PATH", - value=path, - source="conversation_state", - span=None, - confidence=0.6, - ) - ) - if has_user_symbol: - return anchors - if not self._followup.is_follow_up(raw): - return anchors - symbol = state.active_symbol or (state.active_code_span_symbols[0] if state.active_code_span_symbols else None) - if symbol: - anchors.append( - QueryAnchor( - type="SYMBOL", - value=symbol, - source="conversation_state", - span=None, - confidence=0.64, - ) - ) - return anchors - - def _paths_for_carryover(self, active_path_scope: list[str]) -> list[str]: - paths = list(active_path_scope or []) - file_paths = [path for path in paths if self._looks_like_file(path)] - if file_paths: - return file_paths[:1] - return paths[:1] - - def _looks_like_file(self, value: str) -> bool: - tail = (value or "").rsplit("/", 1)[-1] - return "." in tail diff --git a/src/app/modules/agent/intent_router_v2/analysis/docs_query_signals.py b/src/app/modules/agent/intent_router_v2/analysis/docs_query_signals.py deleted file mode 100644 index 1063ff0..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/docs_query_signals.py +++ /dev/null @@ -1,75 +0,0 @@ -from __future__ import annotations - -import re - - -class DocsQuerySignals: - _ENDPOINT_RE = re.compile(r"(?P/(?:[a-z0-9_-]+|{[a-z0-9_]+})(?:/(?:[a-z0-9_-]+|{[a-z0-9_]+}))*)", re.IGNORECASE) - _ENTITY_RE = re.compile(r"(?:entity|сущност[ьи]|модель|объект)\s+(?P[A-Za-z][A-Za-z0-9_-]+(?:\s+[A-Za-z][A-Za-z0-9_-]+)?)", re.IGNORECASE) - _COMPONENT_RE = re.compile(r"(?:component|компонент|service|module|модул[ья])\s+(?P[A-Za-z][A-Za-z0-9_-]+)", re.IGNORECASE) - _WORKFLOW_RE = re.compile(r"(?:workflow|сценарий|процесс|overview)\s+(?P[A-Za-zА-Яа-я0-9_-]+)", re.IGNORECASE) - _DOC_RE = re.compile(r"(?:document|документ|documentation|документац[ияи])\s+(?P[A-Za-zА-Яа-я0-9_.-]+)", re.IGNORECASE) - _CAMEL_RE = re.compile(r"\b(?P[A-Z][A-Za-z0-9]+(?:Manager|Worker|Channel|Service|Module|Status|Health)|[A-Z][a-z0-9]+(?:[A-Z][A-Za-z0-9]+)+)\b") - _QUOTED_RE = re.compile(r"[\"'«“](?P[^\"'»”]{2,})[\"'»”]") - _MULTIWORD_RE = re.compile(r"\b(runtime health|worker status|control plane)\b", re.IGNORECASE) - - def detect_anchor(self, raw: str) -> tuple[str, str | None]: - text = raw or "" - anchor_patterns = [ - ("entity", self._ENTITY_RE), - ("component", self._COMPONENT_RE), - ("workflow", self._WORKFLOW_RE), - ("document", self._DOC_RE), - ] - if not any(ext in text.lower() for ext in (".py", ".ts", ".js", ".java", ".go", ".rb")): - anchor_patterns.insert(0, ("endpoint", self._ENDPOINT_RE)) - for anchor_type, pattern in anchor_patterns: - match = pattern.search(text) - if match: - return anchor_type, str(match.group("value")).strip() - if any(marker in text.lower() for marker in ("api", "endpoint", "topic", "тема")): - return "topic", None - return "none", None - - def has_docs_anchor(self, raw: str) -> bool: - anchor_type, _ = self.detect_anchor(raw) - return anchor_type != "none" - - def query_entity_candidates(self, raw: str) -> list[str]: - text = raw or "" - values: list[str] = [] - for match in self._ENTITY_RE.finditer(text): - values.append(match.group("value").strip()) - for match in self._CAMEL_RE.finditer(text): - values.append(match.group("value").strip()) - for match in self._MULTIWORD_RE.finditer(text): - values.append(match.group(1).strip()) - return self._dedupe(values) - - def query_anchor_candidates(self, raw: str) -> list[str]: - text = raw or "" - values: list[str] = [] - for match in self._ENDPOINT_RE.finditer(text): - values.append(match.group("value").strip()) - for match in self._CAMEL_RE.finditer(text): - values.append(match.group("value").strip()) - for match in self._QUOTED_RE.finditer(text): - values.append(match.group("value").strip()) - return self._dedupe(values) - - def has_component_like_token(self, raw: str) -> bool: - return bool(self._CAMEL_RE.search(raw or "")) - - def _dedupe(self, values: list[str]) -> list[str]: - result: list[str] = [] - seen: set[str] = set() - for value in values: - normalized = value.strip() - if not normalized: - continue - key = normalized.lower() - if key in seen: - continue - seen.add(key) - result.append(normalized) - return result diff --git a/src/app/modules/agent/intent_router_v2/analysis/docs_sub_intent_detector.py b/src/app/modules/agent/intent_router_v2/analysis/docs_sub_intent_detector.py deleted file mode 100644 index ce50f8a..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/docs_sub_intent_detector.py +++ /dev/null @@ -1,161 +0,0 @@ -from __future__ import annotations - -import re - - -class DocsSubIntentDetector: - _ENTITY_LIKE_TOKEN_RE = re.compile(r"\b[A-Z][A-Za-z0-9]+(?:Health|Status|State)\b") - _CAMEL_ENTITY_CONTEXT_RE = re.compile(r"\b[A-Z][a-z0-9]+(?:[A-Z][A-Za-z0-9]+)+\b") - _RELATED_MARKERS = ( - "найди документацию", - "документация по", - "где в документации", - "что связано", - "связанные документы", - "что дальше читать", - "по каким документам идти", - "родительский документ", - "дочерние документы", - "какие документы", - "что еще посмотреть", - "где еще описано", - "где еще используется", - "с чем связан", - "какие страницы связаны", - ) - _FLOW_MARKERS = ( - "цикл", - "как работает", - "lifecycle", - "как происходит", - "как устроен", - "как устроена", - "как проходит", - "последовательность", - "шаги", - "флоу", - "flow", - "workflow", - "сценар", - "процесс", - "по шагам", - ) - _API_MARKERS = ( - "api", - "endpoint", - "method", - "route", - "handler", - "контракт", - "request", - "response", - "webhook", - "метод", - "эндпоинт", - "роут", - ) - _ENTITY_MARKERS = ( - "сущност", - "entity", - "модел", - "бизнес-объект", - "что такое", - "что за", - "объект", - "как используется", - ) - _COMPONENT_MARKERS = ( - "компонент", - "модул", - "сервис", - "класс", - "блок", - "подсистем", - "часть системы", - "роль", - "какую роль", - "manager", - "worker", - "channel", - "service", - "module", - ) - _GENERAL_MARKERS = ( - "что вообще", - "в целом", - "с чего начать", - "обзор", - "какая структура документации", - "какая документация есть", - "что описано", - "что есть в документации", - "обзор документации", - ) - _NATURAL_ENTITY_MARKERS = ("runtime health", "health state", "статус воркера", "состояние воркера", "состояние runtime") - _NATURAL_FLOW_MARKERS = ("health check runtime", "проверка состояния", "как работает health check", "как происходит проверка") - - def detect(self, raw: str, *, intent: str = "DOCUMENTATION_EXPLAIN") -> str: - candidates = self.rank_candidates(raw, intent=intent) - return candidates[0][0] if candidates else "COMPONENT_EXPLAIN" - - def rank_candidates(self, raw: str, *, intent: str = "DOCUMENTATION_EXPLAIN") -> list[tuple[str, int]]: - source = raw or "" - text = " ".join(source.lower().split()) - if intent == "GENERAL_QA": - return [("GENERIC_QA", 100)] - if intent == "OPENAPI_GENERATION": - return [(self._detect_openapi(text), 100)] - if not text: - return [("COMPONENT_EXPLAIN", 1)] - scores = { - "RELATED_DOCS_EXPLAIN": 0, - "API_METHOD_EXPLAIN": 0, - "ENTITY_EXPLAIN": 0, - "COMPONENT_EXPLAIN": 0, - "SYSTEM_FLOW_EXPLAIN": 0, - "GENERIC_QA": 0, - } - if any(marker in text for marker in self._RELATED_MARKERS): - scores["RELATED_DOCS_EXPLAIN"] += 8 - if self._has_http_path(text) or any(marker in text for marker in self._API_MARKERS): - scores["API_METHOD_EXPLAIN"] += 6 - if scores["RELATED_DOCS_EXPLAIN"] > 0 and scores["API_METHOD_EXPLAIN"] > 0: - scores["API_METHOD_EXPLAIN"] -= 2 - if any(marker in text for marker in self._ENTITY_MARKERS): - scores["ENTITY_EXPLAIN"] += 5 - if self._has_entity_like_camel_token(source): - scores["ENTITY_EXPLAIN"] += 3 - if self._looks_like_entity_question(text, source): - scores["ENTITY_EXPLAIN"] += 2 - if any(marker in text for marker in self._NATURAL_ENTITY_MARKERS): - scores["ENTITY_EXPLAIN"] += 3 - if any(marker in text for marker in self._COMPONENT_MARKERS): - scores["COMPONENT_EXPLAIN"] += 5 - if any(marker in text for marker in self._FLOW_MARKERS): - scores["SYSTEM_FLOW_EXPLAIN"] += 5 - if any(marker in text for marker in self._NATURAL_FLOW_MARKERS): - scores["SYSTEM_FLOW_EXPLAIN"] += 3 - if any(marker in text for marker in self._GENERAL_MARKERS): - scores["GENERIC_QA"] += 6 - if not any(scores.values()): - scores["COMPONENT_EXPLAIN"] = 1 - scores["GENERIC_QA"] = 1 - ranked = sorted(scores.items(), key=lambda item: (-item[1], item[0])) - return [item for item in ranked if item[1] > 0] - - def _detect_openapi(self, text: str) -> str: - markers = ("request", "response", "schema", "parameters", "fragment") - if any(marker in text for marker in markers): - return "OPENAPI_FRAGMENT_GENERATE" - return "OPENAPI_METHOD_GENERATE" - - def _has_http_path(self, text: str) -> bool: - return any(token.startswith("/") and len(token) > 1 for token in text.split()) - - def _has_entity_like_camel_token(self, raw: str) -> bool: - return bool(self._ENTITY_LIKE_TOKEN_RE.search(raw or "")) - - def _looks_like_entity_question(self, text: str, raw: str) -> bool: - if not any(marker in text for marker in ("что такое", "что за", "как используется")): - return False - return bool(self._CAMEL_ENTITY_CONTEXT_RE.search(raw or "")) diff --git a/src/app/modules/agent/intent_router_v2/analysis/followup_detector.py b/src/app/modules/agent/intent_router_v2/analysis/followup_detector.py deleted file mode 100644 index d91d662..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/followup_detector.py +++ /dev/null @@ -1,22 +0,0 @@ -from __future__ import annotations - - -class FollowUpDetector: - _MARKERS = ( - "что дальше", - "почему", - "зачем", - "а что", - "уточни", - "подробнее", - "как именно", - "покажи подробнее", - ) - - def is_follow_up(self, raw: str) -> bool: - text = " ".join((raw or "").lower().split()) - if not text: - return False - if len(text.split()) <= 4: - return True - return any(marker in text for marker in self._MARKERS) diff --git a/src/app/modules/agent/intent_router_v2/analysis/keyword_hint_builder.py b/src/app/modules/agent/intent_router_v2/analysis/keyword_hint_builder.py deleted file mode 100644 index 4e6a954..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/keyword_hint_builder.py +++ /dev/null @@ -1,34 +0,0 @@ -from __future__ import annotations - -import re - -from app.modules.agent.intent_router_v2.analysis.normalization import FILE_PATH_RE -from app.modules.agent.intent_router_v2.analysis.symbol_rules import COMMON_PATH_SEGMENTS, PY_KEYWORDS - -_IDENTIFIER_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{2,}") - - -class KeywordHintBuilder: - def build(self, text: str) -> list[str]: - hints: list[str] = [] - path_segments = self._path_segments(text) - for token in _IDENTIFIER_RE.findall(text or ""): - if token.lower() in PY_KEYWORDS: - continue - if token.lower() in path_segments: - continue - if token not in hints: - hints.append(token) - for match in FILE_PATH_RE.finditer(text or ""): - candidate = match.group(0).lower() - if candidate not in hints: - hints.append(candidate) - return hints[:12] - - def _path_segments(self, text: str) -> set[str]: - values: set[str] = set(COMMON_PATH_SEGMENTS) - for match in FILE_PATH_RE.finditer(text or ""): - for part in re.split(r"[/.]+", match.group(0).lower()): - if part: - values.add(part) - return values diff --git a/src/app/modules/agent/intent_router_v2/analysis/keyword_hint_sanitizer.py b/src/app/modules/agent/intent_router_v2/analysis/keyword_hint_sanitizer.py deleted file mode 100644 index f56a517..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/keyword_hint_sanitizer.py +++ /dev/null @@ -1,50 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.models import QueryAnchor - - -class KeywordHintSanitizer: - _GENERIC_KEY_TERMS = {"файл", "класс", "метод", "функция", "документация"} - _DOMAIN_ALLOWLIST = {"RAG", "API", "HTTP", "SQL"} - _DIR_SCOPE_MARKERS = ("в папке", "в директории", "в каталоге") - - def sanitize(self, raw: str, anchors: list[QueryAnchor], base_hints: list[str]) -> list[str]: - text = (raw or "").lower() - allow_dirs = any(marker in text for marker in self._DIR_SCOPE_MARKERS) - file_paths = [anchor.value for anchor in anchors if anchor.type == "FILE_PATH" and self._looks_like_file(anchor.value)] - known_dirs = {path.rsplit("/", 1)[0] for path in file_paths if "/" in path} - result: list[str] = [] - - for anchor in anchors: - if anchor.type == "FILE_PATH": - if self._looks_like_directory(anchor.value): - if not allow_dirs and (known_dirs or file_paths): - continue - self._append(result, anchor.value) - if anchor.type == "SYMBOL": - self._append(result, anchor.value) - - for token in base_hints: - if token in self._DOMAIN_ALLOWLIST: - self._append(result, token) - continue - lowered = token.lower() - if lowered in self._GENERIC_KEY_TERMS: - continue - if token in known_dirs and not allow_dirs: - continue - if "/" in token and "." not in token and not allow_dirs and file_paths: - continue - self._append(result, token) - return result[:8] - - def _append(self, values: list[str], candidate: str) -> None: - if candidate and candidate not in values: - values.append(candidate) - - def _looks_like_file(self, value: str) -> bool: - tail = (value or "").rsplit("/", 1)[-1] - return "." in tail - - def _looks_like_directory(self, value: str) -> bool: - return "/" in (value or "") and not self._looks_like_file(value) diff --git a/src/app/modules/agent/intent_router_v2/analysis/negation_detector.py b/src/app/modules/agent/intent_router_v2/analysis/negation_detector.py deleted file mode 100644 index c3fadeb..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/negation_detector.py +++ /dev/null @@ -1,17 +0,0 @@ -from __future__ import annotations - -import re - -_TEST_NEG_RE = re.compile( - r"(?:не\s+про\s+тест|без\s+тест|кроме\s+тест|про\s+прод\s+код|только\s+прод|production\s+code)", - re.IGNORECASE, -) - - -class NegationDetector: - def detect(self, text: str) -> set[str]: - lowered = (text or "").lower() - negations: set[str] = set() - if _TEST_NEG_RE.search(lowered): - negations.add("tests") - return negations diff --git a/src/app/modules/agent/intent_router_v2/analysis/normalization.py b/src/app/modules/agent/intent_router_v2/analysis/normalization.py deleted file mode 100644 index 133b759..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/normalization.py +++ /dev/null @@ -1,57 +0,0 @@ -from __future__ import annotations - -import re - -CODE_SPAN_RE = re.compile(r"`[^`]*`") -FILE_PATH_RE = re.compile( - r"(? str: - text = raw or "" - protected = _ProtectedText() - text = self._protect(text, protected) - text = self._collapse_whitespace(text) - text = text.translate(QUOTE_TRANSLATION) - text = SPACE_BEFORE_PUNCT_RE.sub(r"\1", text) - text = SPACE_AFTER_PUNCT_RE.sub(r"\1 ", text) - text = self._collapse_whitespace(text) - return protected.restore(text) - - def _protect(self, text: str, protected: "_ProtectedText") -> str: - for pattern in (CODE_SPAN_RE, FILE_PATH_RE, DOTTED_IDENT_RE, CAMEL_RE, SNAKE_RE): - text = pattern.sub(protected.replace, text) - return text - - def _collapse_whitespace(self, text: str) -> str: - return WS_RE.sub(" ", text).strip() - - -class _ProtectedText: - def __init__(self) -> None: - self._items: dict[str, str] = {} - self._index = 0 - - def replace(self, match: re.Match[str]) -> str: - placeholder = f"@@P{self._index}@@" - self._items[placeholder] = match.group(0) - self._index += 1 - return placeholder - - def restore(self, text: str) -> str: - restored = text - for placeholder, value in self._items.items(): - restored = restored.replace(placeholder, value) - return restored diff --git a/src/app/modules/agent/intent_router_v2/analysis/normalization_terms.py b/src/app/modules/agent/intent_router_v2/analysis/normalization_terms.py deleted file mode 100644 index 41aa85e..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/normalization_terms.py +++ /dev/null @@ -1,48 +0,0 @@ -from __future__ import annotations - - -class KeyTermCanonicalizer: - _ALIASES: dict[str, set[str]] = { - "файл": { - "файл", - "файла", - "файле", - "файлу", - "файлом", - "файлы", - "файлов", - "файлам", - "файлами", - }, - "класс": {"класс", "класса", "классе", "классу", "классом", "классы", "классов", "классам"}, - "функция": {"функция", "функции", "функцию", "функцией", "функциях"}, - "метод": {"метод", "метода", "методе", "методу", "методом", "методы"}, - "документация": {"документация", "документации", "документацию"}, - "тест": {"тест", "тесты", "тестов", "тестам", "тестами", "юнит-тест", "юниттест"}, - "модуль": {"модуль", "модуля"}, - "пакет": {"пакет"}, - } - - def __init__(self) -> None: - self._token_to_canonical = self._build_index() - - def canonicalize(self, token: str) -> str | None: - return self._token_to_canonical.get((token or "").lower()) - - def aliases(self) -> set[str]: - values: set[str] = set() - for forms in self._ALIASES.values(): - values.update(forms) - return values - - def is_test_term(self, token: str) -> bool: - canonical = self.canonicalize(token) - return canonical == "тест" - - def _build_index(self) -> dict[str, str]: - index: dict[str, str] = {} - for canonical, forms in self._ALIASES.items(): - index[canonical] = canonical - for form in forms: - index[form] = canonical - return index diff --git a/src/app/modules/agent/intent_router_v2/analysis/query_normalizer.py b/src/app/modules/agent/intent_router_v2/analysis/query_normalizer.py deleted file mode 100644 index 04e3c6c..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/query_normalizer.py +++ /dev/null @@ -1,3 +0,0 @@ -from app.modules.agent.intent_router_v2.analysis.normalization import QueryNormalizer - -__all__ = ["QueryNormalizer"] diff --git a/src/app/modules/agent/intent_router_v2/analysis/query_plan_builder.py b/src/app/modules/agent/intent_router_v2/analysis/query_plan_builder.py deleted file mode 100644 index 8777f19..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/query_plan_builder.py +++ /dev/null @@ -1,319 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.analysis.anchor_extractor import AnchorExtractor -from app.modules.agent.intent_router_v2.analysis.anchor_span_validator import AnchorSpanValidator -from app.modules.agent.intent_router_v2.analysis.conversation_anchor_builder import ConversationAnchorBuilder -from app.modules.agent.intent_router_v2.analysis.docs_sub_intent_detector import DocsSubIntentDetector -from app.modules.agent.intent_router_v2.analysis.keyword_hint_builder import KeywordHintBuilder -from app.modules.agent.intent_router_v2.analysis.keyword_hint_sanitizer import KeywordHintSanitizer -from app.modules.agent.intent_router_v2.models import ConversationState, QueryAnchor, QueryPlan -from app.modules.agent.intent_router_v2.analysis.negation_detector import NegationDetector -from app.modules.agent.intent_router_v2.analysis.normalization import QueryNormalizer -from app.modules.agent.intent_router_v2.analysis.sub_intent_detector import SubIntentDetector -from app.modules.agent.intent_router_v2.analysis.test_signals import has_test_focus, is_negative_test_request, is_test_related_token -from app.modules.agent.intent_router_v2.analysis.term_mapping import RuEnTermMapper - - -class QueryPlanBuilder: - _DOCS_INTENTS = { - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - } - _WHY_MARKERS = ("почему", "зачем", "откуда", "из-за чего") - _NEXT_STEP_MARKERS = ("что дальше", "дальше что", "и что теперь", "продолжай") - _DOCS_TOPIC_HINTS = { - "деплой": ["deploy", "deployment", "docker", "compose", "env", "config", "production", "ci/cd"], - "deploy": ["deploy", "deployment", "docker", "compose", "env", "config", "production", "ci/cd"], - "docker": ["docker", "compose"], - "production": ["production", "deploy", "env"], - "ci": ["ci/cd", "pipeline", "deploy"], - "cd": ["ci/cd", "pipeline", "deploy"], - "установ": ["install", "setup", "config"], - } - def __init__( - self, - normalizer: QueryNormalizer | None = None, - extractor: AnchorExtractor | None = None, - mapper: RuEnTermMapper | None = None, - keyword_hints: KeywordHintBuilder | None = None, - keyword_hint_sanitizer: KeywordHintSanitizer | None = None, - carryover: ConversationAnchorBuilder | None = None, - span_validator: AnchorSpanValidator | None = None, - sub_intent_detector: SubIntentDetector | None = None, - docs_sub_intent_detector: DocsSubIntentDetector | None = None, - negation_detector: NegationDetector | None = None, - ) -> None: - self._normalizer = normalizer or QueryNormalizer() - self._extractor = extractor or AnchorExtractor() - self._mapper = mapper or RuEnTermMapper() - self._keyword_hints_builder = keyword_hints or KeywordHintBuilder() - self._keyword_hint_sanitizer = keyword_hint_sanitizer or KeywordHintSanitizer() - self._carryover = carryover or ConversationAnchorBuilder() - self._span_validator = span_validator or AnchorSpanValidator() - self._sub_intent_detector = sub_intent_detector or SubIntentDetector() - self._docs_sub_intent_detector = docs_sub_intent_detector or DocsSubIntentDetector() - self._negation_detector = negation_detector or NegationDetector() - - def build( - self, - user_query: str, - conversation_state: ConversationState, - continue_mode: bool, - *, - conversation_mode: str = "START", - intent: str = "FALLBACK", - ) -> QueryPlan: - raw = user_query or "" - normalized = self._normalizer.normalize(raw) - if not normalized and raw.strip(): - normalized = raw - negations = self._negation_detector.detect(normalized) - user_anchors = self._span_validator.sanitize(self._extractor.extract(raw), len(raw)) - has_file_path = any(anchor.type == "FILE_PATH" and anchor.source == "user_text" for anchor in user_anchors) - sub_intent = self._sub_intent_detector.detect(raw, has_file_path=has_file_path, negations=negations) - merged_anchors = self._merge_anchors( - raw, - user_anchors, - conversation_state, - continue_mode, - conversation_mode=conversation_mode, - intent=intent, - ) - skip_tests = "tests" in negations or is_negative_test_request(raw) - cleaned_anchors = self._remove_negated_test_terms(skip_tests, merged_anchors) - sub_intent = self._resolve_sub_intent(sub_intent, raw, cleaned_anchors, intent=intent, negations=negations) - if intent in self._DOCS_INTENTS: - sub_intent = self._docs_sub_intent_detector.detect(raw, intent=intent) - elif intent == "FALLBACK": - sub_intent = "GENERIC_QA" - expansions = self._expansions(normalized, cleaned_anchors, skip_tests=skip_tests) - keyword_hints = self._keyword_hints( - raw, - normalized, - cleaned_anchors, - skip_tests=skip_tests, - intent=intent, - state=conversation_state, - ) - symbol_candidates = self._symbol_candidates(cleaned_anchors) - symbol_kind_hint = self._symbol_kind_hint(raw, cleaned_anchors) - path_hints = self._path_hints(cleaned_anchors) - doc_scope_hints = self._doc_scope_hints(intent, raw, keyword_hints, path_hints) - if sub_intent == "OPEN_FILE": - keyword_hints = [] - symbol_candidates = [] - symbol_kind_hint = "unknown" - doc_scope_hints = [] - if intent in self._DOCS_INTENTS: - symbol_candidates = [] - symbol_kind_hint = "unknown" - keyword_hints = self._docs_keyword_hints(raw, keyword_hints) - return QueryPlan( - raw=raw, - normalized=normalized, - sub_intent=sub_intent, - negations=sorted(negations), - expansions=expansions, - keyword_hints=keyword_hints, - path_hints=path_hints, - doc_scope_hints=doc_scope_hints, - symbol_candidates=symbol_candidates, - symbol_kind_hint=symbol_kind_hint, - anchors=cleaned_anchors, - ) - - def _merge_anchors( - self, - raw: str, - anchors: list[QueryAnchor], - state: ConversationState, - continue_mode: bool, - *, - conversation_mode: str, - intent: str, - ) -> list[QueryAnchor]: - has_user_symbol = any(anchor.type == "SYMBOL" and anchor.source == "user_text" for anchor in anchors) - has_user_file = any(anchor.type == "FILE_PATH" and anchor.source == "user_text" for anchor in anchors) - inherited = self._carryover.build( - raw, - state, - continue_mode=continue_mode, - has_user_symbol=has_user_symbol, - has_user_file_path=has_user_file, - ) - if ( - conversation_mode == "SWITCH" - and intent in self._DOCS_INTENTS - and not has_user_file - and not has_user_symbol - and state.active_symbol - ): - inherited.append( - QueryAnchor( - type="SYMBOL", - value=state.active_symbol, - source="conversation_state", - span=None, - confidence=0.62, - ) - ) - return self._dedupe(anchors + inherited) - - def _expansions(self, normalized: str, anchors: list[QueryAnchor], *, skip_tests: bool) -> list[str]: - values = self._mapper.expand(normalized) - has_symbol = any(anchor.type == "SYMBOL" for anchor in anchors) - if has_symbol: - values = [value for value in values if value.lower() not in {"def", "class"}] - if not skip_tests and has_test_focus(normalized): - for candidate in ("test", "unit test"): - if candidate not in values: - values.append(candidate) - for anchor in anchors: - if anchor.type == "SYMBOL" and anchor.value not in values: - values.append(anchor.value) - if skip_tests: - values = [value for value in values if not is_test_related_token(value)] - return values[:16] - - def _keyword_hints( - self, - raw: str, - normalized: str, - anchors: list[QueryAnchor], - *, - skip_tests: bool, - intent: str, - state: ConversationState, - ) -> list[str]: - values = self._keyword_hints_builder.build(normalized) - for anchor in anchors: - if anchor.type not in {"FILE_PATH", "SYMBOL"}: - continue - candidate = anchor.value - if candidate not in values: - values.append(candidate) - if skip_tests: - values = [value for value in values if not is_test_related_token(value)] - sanitized = self._keyword_hint_sanitizer.sanitize(raw, anchors, values) - if intent in self._DOCS_INTENTS and not sanitized: - fallback = list(dict.fromkeys([*self._expansions(normalized, anchors, skip_tests=skip_tests)])) - sanitized = fallback[:3] - if state.active_symbol and state.active_symbol not in sanitized: - sanitized.append(state.active_symbol) - sanitized = sanitized[:5] - return sanitized - - def _remove_negated_test_terms(self, skip_tests: bool, anchors: list[QueryAnchor]) -> list[QueryAnchor]: - if not skip_tests: - return anchors - result: list[QueryAnchor] = [] - for anchor in anchors: - if anchor.type not in {"KEY_TERM", "SYMBOL"}: - result.append(anchor) - continue - if is_test_related_token(anchor.value): - continue - result.append(anchor) - return result - - def _dedupe(self, anchors: list[QueryAnchor]) -> list[QueryAnchor]: - result: list[QueryAnchor] = [] - seen: set[tuple[str, str, str | None, str]] = set() - for anchor in anchors: - key = (anchor.type, anchor.value, anchor.subtype, anchor.source) - if key in seen: - continue - seen.add(key) - result.append(anchor) - return result - - def _resolve_sub_intent( - self, - candidate: str, - raw: str, - anchors: list[QueryAnchor], - *, - intent: str, - negations: set[str], - ) -> str: - if candidate != "EXPLAIN": - return candidate - if intent != "CODE_QA": - return candidate - text = " ".join((raw or "").lower().split()) - has_symbol = any(anchor.type == "SYMBOL" and anchor.confidence >= 0.6 for anchor in anchors) - has_file = any(anchor.type == "FILE_PATH" and self._looks_like_file(anchor.value) and anchor.confidence >= 0.6 for anchor in anchors) - has_user_anchor = any(anchor.source == "user_text" for anchor in anchors) - is_why = any(marker in text for marker in self._WHY_MARKERS) - is_next_steps = any(marker in text for marker in self._NEXT_STEP_MARKERS) - is_short_generic = len(text.split()) <= 4 and text.endswith("?") - if (is_why and has_file and has_symbol) or ((is_next_steps or is_short_generic) and has_file): - return "EXPLAIN_LOCAL" - if "tests" in negations and not has_user_anchor and (has_file or has_symbol): - return "EXPLAIN_LOCAL" - return candidate - - def _looks_like_file(self, value: str) -> bool: - tail = (value or "").rsplit("/", 1)[-1] - return "." in tail - - def _path_hints(self, anchors: list[QueryAnchor]) -> list[str]: - values: list[str] = [] - for anchor in anchors: - if anchor.type != "FILE_PATH": - continue - if anchor.value not in values: - values.append(anchor.value) - return values[:8] - - def _symbol_candidates(self, anchors: list[QueryAnchor]) -> list[str]: - values: list[str] = [] - for anchor in anchors: - if anchor.type != "SYMBOL": - continue - if anchor.value not in values: - values.append(anchor.value) - return values[:8] - - def _symbol_kind_hint(self, raw: str, anchors: list[QueryAnchor]) -> str: - text = " ".join((raw or "").lower().split()) - if "класс" in text or "class " in text: - return "class" - if "функц" in text or "function " in text: - return "function" - if "метод" in text or "method " in text: - return "method" - if "модул" in text or "module " in text: - return "module" - symbol = next((anchor.value for anchor in anchors if anchor.type == "SYMBOL"), "") - if symbol and symbol[:1].isupper(): - return "class" - if "_" in symbol: - return "function" - return "unknown" - - def _docs_keyword_hints(self, raw: str, values: list[str]) -> list[str]: - text = " ".join((raw or "").lower().split()) - result = list(values) - for marker, hints in self._DOCS_TOPIC_HINTS.items(): - if marker not in text: - continue - for hint in hints: - if hint not in result: - result.append(hint) - return result[:12] - - def _doc_scope_hints(self, intent: str, raw: str, keyword_hints: list[str], path_hints: list[str]) -> list[str]: - if intent not in self._DOCS_INTENTS: - return [] - values = list(path_hints) - for candidate in ("README*", "docs/**", "**/*.md"): - if candidate not in values: - values.append(candidate) - for hint in keyword_hints: - if hint.lower() in {"readme", "deploy", "deployment", "docker", "compose"} and hint not in values: - values.append(hint) - if "docs/" in (raw or "").lower() and "docs/**" not in values: - values.append("docs/**") - return values[:8] diff --git a/src/app/modules/agent/intent_router_v2/analysis/sub_intent_detector.py b/src/app/modules/agent/intent_router_v2/analysis/sub_intent_detector.py deleted file mode 100644 index 61b1e90..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/sub_intent_detector.py +++ /dev/null @@ -1,116 +0,0 @@ -from __future__ import annotations - - -class SubIntentDetector: - """Rule-based sub-intent detector. - - Heuristics distinguish two close classes of questions: - - FIND_ENTRYPOINTS: "where does this endpoint/route start?" - - TRACE_FLOW: "what happens step by step for this symbol/request?" - """ - - _OPEN_VERBS = ("открой", "посмотри", "покажи", "выведи", "show", "open") - _EXPLAIN_MARKERS = ("объясни", "как работает", "почему", "что делает", "зачем", "логика", "флоу", "flow") - _TEST_MARKERS = ("тест", "pytest", "unit test", "юнит") - _TRACE_FLOW_MARKERS = ( - "trace flow", - "execution trace", - "dataflow", - "request flow", - "request path", - "поток", - "поток данных", - "поток выполнения", - "поток запроса", - "как проходит запрос", - "как обрабатывается запрос", - "обработка запроса", - "что происходит при запросе", - "какие шаги выполняются", - "по шагам", - "при запуске", - "трассиров", - "кто читает", - "кто пишет", - "что вызывает", - "где создается", - "где создаётся", - ) - _ARCHITECTURE_MARKERS = ( - "архитектур", - "architecture", - "какие сервисы", - "какие обработчики", - "какие компоненты", - "бизнес-логика", - "business logic", - "io", - ) - _ENTRYPOINTS_MARKERS = ( - "entrypoint", - "entrypoints", - "endpoint", - "route", - "handler", - "точки входа", - "точка входа", - "маршрут", - "роут", - "url", - "main", - "запуск", - "как запускается", - "где вход", - ) - _HTTP_ENTRYPOINT_MARKERS = ("endpoint", "route", "handler", "маршрут", "роут", "url") - - def detect(self, raw: str, *, has_file_path: bool, negations: set[str]) -> str: - raw_text = " ".join((raw or "").split()) - text = raw_text.lower() - if not text: - return "EXPLAIN" - if has_file_path and self._has_open_verb(text) and not self._has_explain_markers(text): - return "OPEN_FILE" - if has_file_path and ("файл" in text or "file" in text) and not self._has_explain_markers(text): - return "OPEN_FILE" - if "tests" not in negations and any(marker in text for marker in self._TEST_MARKERS): - return "FIND_TESTS" - has_flow = self._has_trace_flow_markers(text) - has_entrypoints = any(marker in text for marker in self._ENTRYPOINTS_MARKERS) - has_target = has_file_path or self._has_http_path(text) or self._has_symbol_like_target(raw_text) - if has_flow and has_target: - return "TRACE_FLOW" - if self._has_http_path(text) and any(marker in text for marker in self._HTTP_ENTRYPOINT_MARKERS): - return "FIND_ENTRYPOINTS" - if has_entrypoints: - return "FIND_ENTRYPOINTS" - if has_flow: - return "TRACE_FLOW" - if any(marker in text for marker in self._ARCHITECTURE_MARKERS): - return "ARCHITECTURE" - return "EXPLAIN" - - def _has_open_verb(self, text: str) -> bool: - return any(text.startswith(verb) or f" {verb} " in f" {text} " for verb in self._OPEN_VERBS) - - def _has_explain_markers(self, text: str) -> bool: - return any(marker in text for marker in self._EXPLAIN_MARKERS) - - def _has_trace_flow_markers(self, text: str) -> bool: - return any(marker in text for marker in self._TRACE_FLOW_MARKERS) - - def _has_http_path(self, text: str) -> bool: - return any(token.startswith("/") and len(token) > 1 for token in text.split()) - - def _has_symbol_like_target(self, text: str) -> bool: - tokens = [token.strip(".,:;?!()[]{}\"'") for token in text.split()] - for token in tokens: - if not token: - continue - if token.startswith("/"): - return True - if "_" in token: - return True - if len(token) >= 4 and any(char.isupper() for char in token): - return True - return False diff --git a/src/app/modules/agent/intent_router_v2/analysis/symbol_rules.py b/src/app/modules/agent/intent_router_v2/analysis/symbol_rules.py deleted file mode 100644 index e754a49..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/symbol_rules.py +++ /dev/null @@ -1,51 +0,0 @@ -from __future__ import annotations - -PY_KEYWORDS: set[str] = { - "and", - "as", - "assert", - "async", - "await", - "break", - "class", - "continue", - "def", - "del", - "elif", - "else", - "except", - "false", - "finally", - "for", - "from", - "global", - "if", - "import", - "in", - "is", - "lambda", - "none", - "nonlocal", - "not", - "or", - "pass", - "raise", - "return", - "true", - "try", - "while", - "with", - "yield", -} - -COMMON_PATH_SEGMENTS: set[str] = { - "app", - "src", - "docs", - "tests", - "module", - "modules", - "core", - "pkg", - "lib", -} diff --git a/src/app/modules/agent/intent_router_v2/analysis/term_mapping.py b/src/app/modules/agent/intent_router_v2/analysis/term_mapping.py deleted file mode 100644 index 6e18434..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/term_mapping.py +++ /dev/null @@ -1,67 +0,0 @@ -from __future__ import annotations - -import re - -from app.modules.agent.intent_router_v2.analysis.normalization_terms import KeyTermCanonicalizer - -_WORD_RE = re.compile(r"[A-Za-zА-Яа-яЁё-]+") - - -class RuEnTermMapper: - _CANONICAL_MAP = { - "класс": ["class"], - "метод": ["method"], - "функция": ["function", "def"], - "модуль": ["module"], - "пакет": ["package"], - "файл": ["file"], - "тест": ["test", "unit test"], - "документация": ["documentation", "docs"], - "readme": ["readme"], - } - _ENGLISH_SOURCES = { - "class": ["class"], - "method": ["method"], - "function": ["function", "def"], - "module": ["module"], - "package": ["package"], - "file": ["file"], - "test": ["test", "unit test"], - "tests": ["test", "unit test"], - "documentation": ["documentation", "docs"], - "docs": ["documentation", "docs"], - "readme": ["readme"], - "def": ["def"], - } - - def __init__(self, canonicalizer: KeyTermCanonicalizer | None = None) -> None: - self._canonicalizer = canonicalizer or KeyTermCanonicalizer() - - def expand(self, text: str) -> list[str]: - expansions: list[str] = [] - lowered = (text or "").lower() - for token in _WORD_RE.findall(lowered): - canonical = self._canonicalizer.canonicalize(token) or token - self._extend(expansions, self._CANONICAL_MAP.get(canonical, [])) - self._extend(expansions, self._ENGLISH_SOURCES.get(token, [])) - if "unit test" in lowered or "unit tests" in lowered: - self._extend(expansions, self._ENGLISH_SOURCES["test"]) - return expansions - - def key_terms(self) -> tuple[str, ...]: - return tuple(self._CANONICAL_MAP.keys()) - - def all_literal_terms(self) -> tuple[str, ...]: - values = set(self._canonicalizer.aliases()) - values.update(self._CANONICAL_MAP.keys()) - values.update(self._ENGLISH_SOURCES.keys()) - for targets in self._CANONICAL_MAP.values(): - values.update(target.lower() for target in targets) - for targets in self._ENGLISH_SOURCES.values(): - values.update(target.lower() for target in targets) - return tuple(sorted(values)) - - def _extend(self, result: list[str], values: list[str]) -> None: - for value in values: - if value not in result: - result.append(value) diff --git a/src/app/modules/agent/intent_router_v2/analysis/test_signals.py b/src/app/modules/agent/intent_router_v2/analysis/test_signals.py deleted file mode 100644 index 60c4a52..0000000 --- a/src/app/modules/agent/intent_router_v2/analysis/test_signals.py +++ /dev/null @@ -1,40 +0,0 @@ -from __future__ import annotations - -import re - -_NEGATIVE_TEST_RE = re.compile(r"\b(?:не|без|кроме)\b[^.?!]{0,28}\bтест", re.IGNORECASE) -_NEGATIVE_TEST_MARKERS = ("не про тест", "без тест", "кроме тест", "про прод код", "только прод", "production code") -_POSITIVE_TEST_MARKERS = ( - "тест", - "tests", - "pytest", - "unit test", - "unit tests", - "тестиру", -) -_TEST_TERMS = {"тест", "тесты", "test", "tests", "pytest", "unit", "unit test", "юнит-тест", "юниттест"} - - -def is_negative_test_request(text: str) -> bool: - lowered = (text or "").lower() - if _NEGATIVE_TEST_RE.search(lowered): - return True - return any(marker in lowered for marker in _NEGATIVE_TEST_MARKERS) - - -def has_test_focus(text: str) -> bool: - lowered = (text or "").lower() - if is_negative_test_request(lowered): - return False - return any(marker in lowered for marker in _POSITIVE_TEST_MARKERS) - - -def is_test_related_token(value: str) -> bool: - lowered = (value or "").lower().strip() - if not lowered: - return False - if lowered in _TEST_TERMS: - return True - if lowered.startswith("test"): - return True - return lowered.startswith("тест") diff --git a/src/app/modules/agent/intent_router_v2/factory.py b/src/app/modules/agent/intent_router_v2/factory.py deleted file mode 100644 index 6a5c65b..0000000 --- a/src/app/modules/agent/intent_router_v2/factory.py +++ /dev/null @@ -1,27 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.llm import AgentLlmService -from app.modules.agent.llm.prompt_loader import PromptLoader -from app.modules.agent.intent_router_v2.intent.classifier import IntentClassifierV2 -from app.modules.agent.intent_router_v2.intent.llm_disambiguator import DocsLlmDisambiguator -from app.modules.agent.intent_router_v2.router import IntentRouterV2 -from app.modules.shared.env_loader import load_workspace_env -from app.modules.shared.gigachat.client import GigaChatClient -from app.modules.shared.gigachat.settings import GigaChatSettings -from app.modules.shared.gigachat.token_provider import GigaChatTokenProvider - - -class GigaChatIntentRouterFactory: - def build(self) -> IntentRouterV2: - load_workspace_env() - settings = GigaChatSettings.from_env() - token_provider = GigaChatTokenProvider(settings) - client = GigaChatClient(settings, token_provider) - prompt_loader = PromptLoader() - llm = AgentLlmService(client=client, prompts=prompt_loader) - classifier = IntentClassifierV2(llm=llm) - return IntentRouterV2( - classifier=classifier, - llm_disambiguator=DocsLlmDisambiguator(llm), - enable_llm_disambiguation=True, - ) diff --git a/src/app/modules/agent/intent_router_v2/intent/__init__.py b/src/app/modules/agent/intent_router_v2/intent/__init__.py deleted file mode 100644 index a3fb94c..0000000 --- a/src/app/modules/agent/intent_router_v2/intent/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -from app.modules.agent.intent_router_v2.intent.classifier import IntentClassifierV2 -from app.modules.agent.intent_router_v2.intent.conversation_policy import ConversationPolicy -from app.modules.agent.intent_router_v2.intent.graph_id_resolver import GraphIdResolver - -__all__ = ["IntentClassifierV2", "ConversationPolicy", "GraphIdResolver"] diff --git a/src/app/modules/agent/intent_router_v2/intent/classifier.py b/src/app/modules/agent/intent_router_v2/intent/classifier.py deleted file mode 100644 index 380925a..0000000 --- a/src/app/modules/agent/intent_router_v2/intent/classifier.py +++ /dev/null @@ -1,248 +0,0 @@ -from __future__ import annotations - -import json -import re - -from app.modules.agent.intent_router_v2.analysis.docs_query_signals import DocsQuerySignals -from app.modules.agent.intent_router_v2.models import ConversationState, IntentDecision -from app.modules.agent.intent_router_v2.protocols import TextGenerator -from app.modules.agent.intent_router_v2.analysis.test_signals import has_test_focus - -_CODE_FILE_PATH_RE = re.compile( - r"\b(?:[\w.-]+/)*[\w.-]+\.(?:py|js|jsx|ts|tsx|java|kt|go|rb|php|c|cc|cpp|h|hpp|cs|swift|rs)(?!\w)\b", - re.IGNORECASE, -) - - -class IntentClassifierV2: - _GENERATE_DOCS_MARKERS = ( - "сгенерируй документац", - "подготовь документац", - "создай документац", - "генерац", - "generate documentation", - "write documentation", - ) - _DOCS_MARKERS = ("документац", "readme", "docs/", ".md", "spec", "runbook", "markdown") - _OPENAPI_MARKERS = ("openapi", "swagger", "yaml", "spec", "schema") - _OPENAPI_EXTENDED_MARKERS = ("request body", "response schema", "request schema", "response body") - _DOCS_RELATED_MARKERS = ( - "что связано", - "связанные документы", - "что дальше читать", - "по каким документам идти", - "родительский документ", - "дочерние документы", - "какие документы", - "что еще посмотреть", - "где еще описано", - "где еще используется", - "с чем связан", - "какие страницы связаны", - "документация по", - "найди", - "где описан", - "где в документации", - "покажи документы", - "после этого", - ) - _DOCS_EXPLAIN_MARKERS = ("объясни", "как работает", "что делает", "что такое") - _DOCS_FLOW_MARKERS = ( - "цикл", - "сценар", - "процесс", - "происходит", - "последовательность", - "шаги", - "как работает процесс", - "как происходит", - "как устроен процесс", - "workflow", - "flow", - "lifecycle", - "жизненный цикл", - ) - _DOCS_COMPONENT_MARKERS = ( - "компонент", - "модул", - "подсистем", - "часть системы", - "блок", - "роль", - "какую роль", - "что делает", - "как работает компонент", - "как устроен", - ) - _DOCS_ENTITY_MARKERS = ("сущност", "entity", "бизнес-объект", "объект", "как используется", "что такое") - _DOCS_API_MARKERS = ("api", "endpoint", "эндпоинт", "ручк", "request", "response") - _GENERAL_QA_MARKERS = ( - "помоги понять", - "с чего начать", - "как начать", - "куда смотреть", - "что тут важно", - "что вообще", - "в целом", - "обзор", - "какая структура документации", - "структура документации", - "какая документация есть", - "что описано в документации", - "что есть по сервису", - ) - _CODE_MARKERS = ( - "по коду", - "код", - "класс", - "метод", - "функц", - "модул", - "пакет", - "файл", - "block", - "блок", - "handler", - "endpoint", - "архитектур", - "сервис", - "обработчик", - "pipeline", - "бизнес-логика", - ) - - def __init__(self, llm: TextGenerator | None = None) -> None: - self._llm = llm - self._docs_signals = DocsQuerySignals() - - def classify(self, user_query: str, conversation_state: ConversationState) -> IntentDecision: - deterministic = self._deterministic(user_query) - if deterministic: - return deterministic - llm_decision = self._classify_with_llm(user_query, conversation_state) - if llm_decision: - return llm_decision - return IntentDecision(intent="FALLBACK", confidence=0.55, reason="fallback") - - def _deterministic(self, user_query: str) -> IntentDecision | None: - text = " ".join((user_query or "").lower().split()) - if any(marker in text for marker in self._GENERATE_DOCS_MARKERS): - return IntentDecision(intent="GENERATE_DOCS_FROM_CODE", confidence=0.97, reason="deterministic_generate_docs") - if any(marker in text for marker in (*self._OPENAPI_MARKERS, *self._OPENAPI_EXTENDED_MARKERS)): - return IntentDecision(intent="OPENAPI_GENERATION", confidence=0.98, reason="deterministic_openapi") - if self._is_general_docs_qa(text): - return IntentDecision(intent="GENERAL_QA", confidence=0.76, reason="deterministic_general_docs") - if self._is_docs_explain(text, user_query): - return IntentDecision(intent="DOCUMENTATION_EXPLAIN", confidence=0.91, reason="deterministic_docs_explain") - if self._is_related_docs(text): - return IntentDecision(intent="DOCUMENTATION_EXPLAIN", confidence=0.9, reason="deterministic_docs_related") - if not has_test_focus(text) and self._docs_signals.has_component_like_token(user_query) and not self._is_general_docs_qa(text): - return IntentDecision(intent="DOCUMENTATION_EXPLAIN", confidence=0.82, reason="deterministic_docs_component_anchor") - if self._looks_like_docs_question(text): - return IntentDecision(intent="DOCUMENTATION_EXPLAIN", confidence=0.9, reason="deterministic_docs") - if self._looks_like_code_question(user_query, text): - return IntentDecision(intent="CODE_QA", confidence=0.9, reason="deterministic_code") - return IntentDecision(intent="GENERAL_QA", confidence=0.62, reason="deterministic_general") - - def _classify_with_llm(self, user_query: str, conversation_state: ConversationState) -> IntentDecision | None: - if self._llm is None: - return None - payload = json.dumps( - { - "message": user_query, - "active_intent": conversation_state.active_intent, - "last_query": conversation_state.last_query, - "allowed_intents": [ - "CODE_QA", - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - "GENERATE_DOCS_FROM_CODE", - ], - }, - ensure_ascii=False, - ) - try: - raw = self._llm.generate("rag_intent_router_v2", payload, log_context="rag.intent_router_v2.classify").strip() - except Exception: - return None - parsed = self._parse(raw) - if parsed is None: - return None - return parsed - - def _parse(self, raw: str) -> IntentDecision | None: - candidate = self._strip_code_fence(raw) - try: - payload = json.loads(candidate) - except json.JSONDecodeError: - return None - intent = str(payload.get("intent") or "").strip().upper() - if intent not in { - "CODE_QA", - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - "GENERATE_DOCS_FROM_CODE", - "FALLBACK", - }: - return None - sub_intent = str(payload.get("sub_intent") or "").strip() or None - return IntentDecision( - intent=intent, - sub_intent=sub_intent, - confidence=float(payload.get("confidence") or 0.7), - reason=str(payload.get("reason") or "llm").strip() or "llm", - ) - - def _strip_code_fence(self, text: str) -> str: - if not text.startswith("```"): - return text - lines = text.splitlines() - if len(lines) < 3 or lines[-1].strip() != "```": - return text - return "\n".join(lines[1:-1]).strip() - - def _looks_like_docs_question(self, text: str) -> bool: - if self._has_code_file_path(text): - return False - if self._is_general_docs_qa(text): - return False - return self._has_docs_context(text) or self._has_docs_subject(text) - - def _has_docs_context(self, text: str) -> bool: - return any(marker in text for marker in self._DOCS_MARKERS) - - def _is_docs_discovery(self, text: str) -> bool: - return self._is_related_docs(text) - - def _is_related_docs(self, text: str) -> bool: - return any(marker in text for marker in self._DOCS_RELATED_MARKERS) - - def _is_docs_explain(self, text: str, user_query: str) -> bool: - if not any(marker in text for marker in self._DOCS_EXPLAIN_MARKERS): - return False - return self._docs_signals.has_docs_anchor(user_query) or self._has_docs_subject(text) - - def _has_docs_subject(self, text: str) -> bool: - return any( - marker in text - for marker in (*self._DOCS_FLOW_MARKERS, *self._DOCS_COMPONENT_MARKERS, *self._DOCS_ENTITY_MARKERS, *self._DOCS_API_MARKERS) - ) - - def _is_general_docs_qa(self, text: str) -> bool: - return any(marker in text for marker in self._GENERAL_QA_MARKERS) - - def _looks_like_code_question(self, raw_text: str, lowered: str) -> bool: - if self._has_code_file_path(raw_text): - return True - if has_test_focus(lowered): - return True - if any(marker in lowered for marker in self._DOCS_MARKERS) and not any(marker in lowered for marker in self._CODE_MARKERS): - return False - if any(marker in lowered for marker in self._CODE_MARKERS): - return True - return bool(re.search(r"\b[a-z_][A-Za-z0-9_]{2,}\(", raw_text or "")) - - def _has_code_file_path(self, text: str) -> bool: - return bool(_CODE_FILE_PATH_RE.search(text or "")) diff --git a/src/app/modules/agent/intent_router_v2/intent/conversation_policy.py b/src/app/modules/agent/intent_router_v2/intent/conversation_policy.py deleted file mode 100644 index b32d16e..0000000 --- a/src/app/modules/agent/intent_router_v2/intent/conversation_policy.py +++ /dev/null @@ -1,73 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.models import ConversationState, IntentDecision - - -class ConversationPolicy: - _SWITCH_MARKERS = ( - "переключ", - "new task", - "switch to", - "instead", - ) - _FOLLOWUP_MARKERS = ( - "теперь", - "а теперь", - "давай теперь", - "дальше", - "потом", - "следом", - "next", - "then", - ) - _DOCS_SIGNALS = ("документац", "readme", "docs/", ".md") - _CODE_SIGNALS = ("по коду", "класс", "метод", "файл", "блок кода", "function", "class") - - def resolve(self, decision: IntentDecision, user_query: str, conversation_state: ConversationState) -> tuple[str, str]: - active_intent = conversation_state.active_intent - if active_intent is None: - return decision.intent, "START" - followup_likely = self._is_followup_likely(user_query) - if active_intent == decision.intent: - return active_intent, "FOLLOWUP_LIKELY" if followup_likely else "CONTINUE" - if self._has_explicit_switch(user_query): - return decision.intent, "SWITCH" - if self._is_hard_mismatch(active_intent, decision.intent, user_query): - return decision.intent, "SWITCH" - if followup_likely: - return active_intent, "FOLLOWUP_LIKELY" - return active_intent, "CONTINUE" - - def _has_explicit_switch(self, user_query: str) -> bool: - text = " ".join((user_query or "").lower().split()) - return any(marker in text for marker in self._SWITCH_MARKERS) - - def _is_followup_likely(self, user_query: str) -> bool: - text = " ".join((user_query or "").lower().split()) - return any(marker in text for marker in self._FOLLOWUP_MARKERS) - - def _is_hard_mismatch(self, active_intent: str, candidate_intent: str, user_query: str) -> bool: - if active_intent == candidate_intent: - return False - text = " ".join((user_query or "").lower().split()) - if candidate_intent == "GENERATE_DOCS_FROM_CODE": - return True - if candidate_intent in { - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - "DOCUMENTATION_DISCOVERY", - "DOCUMENTATION_NAVIGATION", - "OPENAPI_FROM_DOCUMENTATION", - }: - return any(signal in text for signal in self._DOCS_SIGNALS) - if candidate_intent == "CODE_QA" and active_intent in { - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - "DOCUMENTATION_DISCOVERY", - "DOCUMENTATION_NAVIGATION", - "OPENAPI_FROM_DOCUMENTATION", - }: - return any(signal in text for signal in self._CODE_SIGNALS) - return False diff --git a/src/app/modules/agent/intent_router_v2/intent/graph_id_resolver.py b/src/app/modules/agent/intent_router_v2/intent/graph_id_resolver.py deleted file mode 100644 index 47a5dcf..0000000 --- a/src/app/modules/agent/intent_router_v2/intent/graph_id_resolver.py +++ /dev/null @@ -1,18 +0,0 @@ -from __future__ import annotations - - -class GraphIdResolver: - _GRAPH_MAP = { - "CODE_QA": "CodeQAGraph", - "DOCUMENTATION_EXPLAIN": "DocsQAGraph", - "OPENAPI_GENERATION": "DocsQAGraph", - "GENERAL_QA": "DocsQAGraph", - "DOCUMENTATION_DISCOVERY": "DocsQAGraph", - "DOCUMENTATION_NAVIGATION": "DocsQAGraph", - "OPENAPI_FROM_DOCUMENTATION": "DocsQAGraph", - "GENERATE_DOCS_FROM_CODE": "GenerateDocsFromCodeGraph", - "FALLBACK": "DocsQAGraph", - } - - def resolve(self, intent: str) -> str: - return self._GRAPH_MAP[intent] diff --git a/src/app/modules/agent/intent_router_v2/intent/llm_disambiguator.py b/src/app/modules/agent/intent_router_v2/intent/llm_disambiguator.py deleted file mode 100644 index 871a20c..0000000 --- a/src/app/modules/agent/intent_router_v2/intent/llm_disambiguator.py +++ /dev/null @@ -1,55 +0,0 @@ -from __future__ import annotations - -import json - -from app.modules.agent.intent_router_v2.protocols import TextGenerator - - -class DocsLlmDisambiguator: - _ALLOWED = { - "SYSTEM_FLOW_EXPLAIN", - "COMPONENT_EXPLAIN", - "API_METHOD_EXPLAIN", - "ENTITY_EXPLAIN", - "RELATED_DOCS_EXPLAIN", - "GENERIC_QA", - "OPENAPI_METHOD_GENERATE", - "OPENAPI_FRAGMENT_GENERATE", - } - - def __init__(self, llm: TextGenerator) -> None: - self._llm = llm - - def choose(self, payload: dict[str, object]) -> dict[str, str] | None: - raw = self._llm.generate( - "rag_docs_router_disambiguation_v1", - json.dumps(payload, ensure_ascii=False), - log_context="rag.intent_router_v2.disambiguate", - ).strip() - parsed = self._parse(raw) - if parsed is None: - return None - return parsed - - def _parse(self, raw: str) -> dict[str, str] | None: - candidate = self._strip_code_fence(raw) - try: - payload = json.loads(candidate) - except json.JSONDecodeError: - return None - sub_intent = str(payload.get("sub_intent") or "").strip() - if sub_intent not in self._ALLOWED: - return None - return { - "sub_intent": sub_intent, - "reason": str(payload.get("reason") or "").strip(), - "confidence": str(payload.get("confidence") or "").strip(), - } - - def _strip_code_fence(self, text: str) -> str: - if not text.startswith("```"): - return text - lines = text.splitlines() - if len(lines) >= 3 and lines[-1].strip() == "```": - return "\n".join(lines[1:-1]).strip() - return text diff --git a/src/app/modules/agent/intent_router_v2/local_runner.py b/src/app/modules/agent/intent_router_v2/local_runner.py deleted file mode 100644 index dcc0c3f..0000000 --- a/src/app/modules/agent/intent_router_v2/local_runner.py +++ /dev/null @@ -1,25 +0,0 @@ -from __future__ import annotations - -import logging - -from app.modules.agent.intent_router_v2.models import ConversationState, IntentRouterResult, RepoContext -from app.modules.agent.intent_router_v2.router import IntentRouterV2 - -LOGGER = logging.getLogger(__name__) - - -class IntentRouterScenarioRunner: - def __init__(self, router: IntentRouterV2) -> None: - self._router = router - - def run(self, queries: list[str], repo_context: RepoContext | None = None) -> list[IntentRouterResult]: - state = ConversationState() - context = repo_context or RepoContext() - results: list[IntentRouterResult] = [] - for index, user_query in enumerate(queries, start=1): - LOGGER.warning("intent router local input: turn=%s user_query=%s", index, user_query) - result = self._router.route(user_query, state, context) - LOGGER.warning("intent router local output: turn=%s result=%s", index, result.model_dump_json(ensure_ascii=False)) - results.append(result) - state = state.advance(result) - return results diff --git a/src/app/modules/agent/intent_router_v2/logger.py b/src/app/modules/agent/intent_router_v2/logger.py deleted file mode 100644 index e3fbb72..0000000 --- a/src/app/modules/agent/intent_router_v2/logger.py +++ /dev/null @@ -1,24 +0,0 @@ -from __future__ import annotations - -import json -import logging - -from app.modules.agent.intent_router_v2.models import ConversationState, IntentRouterResult, RepoContext - -LOGGER = logging.getLogger(__name__) - - -class IntentRouterLogger: - def log_request(self, user_query: str, conversation_state: ConversationState, repo_context: RepoContext) -> None: - LOGGER.warning( - "intent router v2 request: turn=%s active_intent=%s user_query=%s languages=%s domains=%s", - conversation_state.turn_index + 1, - conversation_state.active_intent, - " ".join((user_query or "").split()), - repo_context.languages, - repo_context.available_domains, - ) - - def log_result(self, result: IntentRouterResult) -> None: - payload = json.dumps(result.model_dump(), ensure_ascii=False) - LOGGER.warning("intent router v2 result: %s", payload) diff --git a/src/app/modules/agent/intent_router_v2/models.py b/src/app/modules/agent/intent_router_v2/models.py deleted file mode 100644 index 53b0860..0000000 --- a/src/app/modules/agent/intent_router_v2/models.py +++ /dev/null @@ -1,250 +0,0 @@ -from __future__ import annotations - -import re -from typing import Literal - -from pydantic import BaseModel, ConfigDict, Field, field_validator - - -IntentType = Literal[ - "CODE_QA", - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - "GENERATE_DOCS_FROM_CODE", - "FALLBACK", # deprecated alias, prefer GENERAL_QA - "DOCUMENTATION_DISCOVERY", # deprecated alias, prefer DOCUMENTATION_EXPLAIN - "DOCUMENTATION_NAVIGATION", # deprecated alias, prefer DOCUMENTATION_EXPLAIN - "OPENAPI_FROM_DOCUMENTATION", # deprecated alias, prefer OPENAPI_GENERATION -] -ConversationMode = Literal["START", "CONTINUE", "SWITCH", "FOLLOWUP_LIKELY"] -RetrievalProfile = Literal["code", "docs", "fallback"] -AnchorType = Literal["FILE_PATH", "SYMBOL", "DOC_REF", "KEY_TERM"] -AnchorSource = Literal["user_text", "conversation_state", "heuristic"] -SymbolKindHint = Literal["class", "function", "method", "module", "unknown"] -SymbolResolutionStatus = Literal["not_requested", "pending", "resolved", "ambiguous", "not_found"] -MatchedAnchorType = Literal["endpoint", "entity", "component", "workflow", "topic", "document", "none"] -MatchedIntentSource = Literal["deterministic", "llm"] -_INLINE_CODE_RE = re.compile(r"`([^`]*)`") -_CODE_SYMBOL_RE = re.compile(r"\b([A-Za-z_][A-Za-z0-9_]{2,})\b") - - -class AnchorSpan(BaseModel): - model_config = ConfigDict(extra="forbid") - - start: int = 0 - end: int = 0 - - -class QueryAnchor(BaseModel): - model_config = ConfigDict(extra="forbid") - - type: AnchorType - value: str - source: AnchorSource = "user_text" - subtype: str | None = None - span: AnchorSpan | None = None - confidence: float = 0.0 - - @field_validator("confidence") - @classmethod - def clamp_confidence(cls, value: float) -> float: - return max(0.0, min(1.0, float(value))) - - -class QueryPlan(BaseModel): - model_config = ConfigDict(extra="forbid") - - raw: str - normalized: str - sub_intent: str = "EXPLAIN" - negations: list[str] = Field(default_factory=list) - expansions: list[str] = Field(default_factory=list) - keyword_hints: list[str] = Field(default_factory=list) - path_hints: list[str] = Field(default_factory=list) - doc_scope_hints: list[str] = Field(default_factory=list) - symbol_candidates: list[str] = Field(default_factory=list) - symbol_kind_hint: SymbolKindHint = "unknown" - anchors: list[QueryAnchor] = Field(default_factory=list) - - -class FuzzySymbolSearch(BaseModel): - model_config = ConfigDict(extra="forbid") - - enabled: bool = True - max_distance: int = 2 - top_k: int = 5 - - -class RetrievalConstraints(BaseModel): - model_config = ConfigDict(extra="forbid") - - include_globs: list[str] = Field(default_factory=list) - exclude_globs: list[str] = Field(default_factory=list) - prefer_globs: list[str] = Field(default_factory=list) - test_file_globs: list[str] = Field(default_factory=list) - test_symbol_patterns: list[str] = Field(default_factory=list) - max_candidates: int = 20 - fuzzy_symbol_search: FuzzySymbolSearch = Field(default_factory=FuzzySymbolSearch) - - -class SymbolResolution(BaseModel): - model_config = ConfigDict(extra="forbid") - - status: SymbolResolutionStatus = "not_requested" - resolved_symbol: str | None = None - alternatives: list[str] = Field(default_factory=list) - confidence: float = 0.0 - - -class LayerQuery(BaseModel): - model_config = ConfigDict(extra="forbid") - - layer_id: str - top_k: int - - -class CodeRetrievalFilters(BaseModel): - model_config = ConfigDict(extra="forbid") - - test_policy: str = "EXCLUDE" - path_scope: list[str] = Field(default_factory=list) - language: list[str] = Field(default_factory=list) - - -class DocsRetrievalFilters(BaseModel): - model_config = ConfigDict(extra="forbid") - - path_scope: list[str] = Field(default_factory=list) - doc_kinds: list[str] = Field(default_factory=list) - doc_language: list[str] = Field(default_factory=list) - doc_type: str | None = None - - -class HybridRetrievalFilters(BaseModel): - model_config = ConfigDict(extra="forbid") - - test_policy: str = "EXCLUDE" - path_scope: list[str] = Field(default_factory=list) - language: list[str] = Field(default_factory=list) - doc_kinds: list[str] = Field(default_factory=list) - doc_language: list[str] = Field(default_factory=list) - doc_type: str | None = None - - -class RetrievalSpec(BaseModel): - model_config = ConfigDict(extra="forbid") - - domains: list[str] = Field(default_factory=list) - layer_queries: list[LayerQuery] = Field(default_factory=list) - filters: CodeRetrievalFilters | DocsRetrievalFilters | HybridRetrievalFilters = Field(default_factory=CodeRetrievalFilters) - rerank_profile: str = "" - - -class EvidencePolicy(BaseModel): - model_config = ConfigDict(extra="forbid") - - require_def: bool = False - require_flow: bool = False - require_spec: bool = False - allow_answer_without_evidence: bool = False - - -class IntentRouterResult(BaseModel): - model_config = ConfigDict(extra="forbid") - - schema_version: str = "1.1" - intent: IntentType - retrieval_profile: RetrievalProfile = "code" - graph_id: str - conversation_mode: ConversationMode - query_plan: QueryPlan - retrieval_spec: RetrievalSpec - retrieval_constraints: RetrievalConstraints = Field(default_factory=RetrievalConstraints) - symbol_resolution: SymbolResolution = Field(default_factory=SymbolResolution) - evidence_policy: EvidencePolicy - matched_anchor_type: MatchedAnchorType = "none" - matched_anchor_value: str | None = None - matched_intent_source: MatchedIntentSource = "deterministic" - routing_reason: str = "" - routing_mode: str = "deterministic" - is_ambiguous: bool = False - ambiguity_reason: str = "" - deterministic_candidates: list[str] = Field(default_factory=list) - deterministic_selected_sub_intent: str = "" - llm_router_used: bool = False - llm_router_selected_sub_intent: str = "" - llm_router_reason: str = "" - llm_router_confidence: str = "" - llm_router_error: str = "" - - -class ConversationState(BaseModel): - model_config = ConfigDict(extra="forbid") - - active_intent: IntentType | None = None - active_domain: str | None = None - active_anchors: list[QueryAnchor] = Field(default_factory=list) - active_symbol: str | None = None - active_path_scope: list[str] = Field(default_factory=list) - active_code_span_symbols: list[str] = Field(default_factory=list) - last_query: str = "" - turn_index: int = 0 - - def advance(self, result: IntentRouterResult) -> "ConversationState": - user_anchors = [anchor for anchor in result.query_plan.anchors if anchor.source == "user_text"] - symbol_candidates = [anchor.value for anchor in user_anchors if anchor.type == "SYMBOL"] - has_user_file_anchor = any(anchor.type == "FILE_PATH" for anchor in user_anchors) - if symbol_candidates: - active_symbol = symbol_candidates[-1] - elif has_user_file_anchor: - active_symbol = None - else: - active_symbol = self.active_symbol - raw_code_symbols = _extract_code_symbols(result.query_plan.raw) - active_code_span_symbols = raw_code_symbols or list(self.active_code_span_symbols) - path_scope = list(getattr(result.retrieval_spec.filters, "path_scope", []) or []) - active_domains = list(result.retrieval_spec.domains or []) - active_domain = active_domains[0] if len(active_domains) == 1 else self.active_domain - return ConversationState( - active_intent=result.intent, - active_domain=active_domain, - active_anchors=list(user_anchors), - active_symbol=active_symbol, - active_path_scope=path_scope or list(self.active_path_scope), - active_code_span_symbols=active_code_span_symbols, - last_query=result.query_plan.raw, - turn_index=self.turn_index + 1, - ) - - -class RepoContext(BaseModel): - model_config = ConfigDict(extra="forbid") - - languages: list[str] = Field(default_factory=list) - available_domains: list[str] = Field(default_factory=lambda: ["CODE", "DOCS"]) - available_layers: list[str] = Field(default_factory=list) - - -class IntentDecision(BaseModel): - model_config = ConfigDict(extra="forbid") - - intent: IntentType - sub_intent: str | None = None - confidence: float = 0.0 - reason: str = "" - - @field_validator("confidence") - @classmethod - def clamp_confidence(cls, value: float) -> float: - return max(0.0, min(1.0, float(value))) - - -def _extract_code_symbols(raw: str) -> list[str]: - symbols: list[str] = [] - for match in _INLINE_CODE_RE.finditer(raw or ""): - snippet = match.group(1) - for token in _CODE_SYMBOL_RE.findall(snippet): - if token not in symbols: - symbols.append(token) - return symbols[:8] diff --git a/src/app/modules/agent/intent_router_v2/protocols.py b/src/app/modules/agent/intent_router_v2/protocols.py deleted file mode 100644 index 4b88f20..0000000 --- a/src/app/modules/agent/intent_router_v2/protocols.py +++ /dev/null @@ -1,7 +0,0 @@ -from __future__ import annotations - -from typing import Protocol - - -class TextGenerator(Protocol): - def generate(self, prompt_name: str, user_input: str, *, log_context: str | None = None) -> str: ... diff --git a/src/app/modules/agent/intent_router_v2/readme.md b/src/app/modules/agent/intent_router_v2/readme.md deleted file mode 100644 index 50ef330..0000000 --- a/src/app/modules/agent/intent_router_v2/readme.md +++ /dev/null @@ -1,37 +0,0 @@ -# intent_router_v2 - -## Назначение -Маршрутизация пользовательского запроса в retrieval-профиль и граф выполнения. - -## Структура пакета -- `router.py` — фасад `IntentRouterV2`. -- `intent/` — классификация intent и политика диалога. -- `analysis/` — нормализация запроса, anchors, sub-intent, план запроса. -- `retrieval/` — выбор слоев RAG и фильтров retrieval. -- `factory.py` — сборка роутера с GigaChat-классификатором. - -## Публичный API -- `IntentRouterV2.route(user_query, conversation_state, repo_context) -> IntentRouterResult` -- `GigaChatIntentRouterFactory.build() -> IntentRouterV2` -- `IntentRouterScenarioRunner.run(queries, repo_context) -> list[IntentRouterResult]` -- `models`: - - `ConversationState` - - `RepoContext` - - `IntentRouterResult` - - `QueryPlan` - - `QueryAnchor` - -## Контракт результата `IntentRouterResult` -- `intent`: `CODE_QA | DOCUMENTATION_EXPLAIN | GENERATE_DOCS_FROM_CODE | FALLBACK` -- `retrieval_profile`: `code | docs | fallback` -- `graph_id`: целевой graph в agent runtime -- `query_plan`: нормализованный запрос, anchors, sub-intent, keyword/path/doc hints -- `retrieval_spec`: слои + фильтры для RAG -- `retrieval_constraints`: include/exclude ограничения -- `symbol_resolution`: начальный статус резолва символа -- `evidence_policy`: политика набора источников - -## Правила расширения -- Новая логика анализа запроса добавляется в `analysis/`. -- Новая политика retrieval добавляется в `retrieval/`. -- `router.py` остается orchestration-слоем и не содержит доменной логики парсинга. diff --git a/src/app/modules/agent/intent_router_v2/retrieval_planning/__init__.py b/src/app/modules/agent/intent_router_v2/retrieval_planning/__init__.py deleted file mode 100644 index a3764ae..0000000 --- a/src/app/modules/agent/intent_router_v2/retrieval_planning/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -from app.modules.agent.intent_router_v2.retrieval_planning.retrieval_spec_factory import RetrievalSpecFactory -from app.modules.agent.intent_router_v2.retrieval_planning.retrieval_constraints_factory import RetrievalConstraintsFactory - -__all__ = ["RetrievalSpecFactory", "RetrievalConstraintsFactory"] diff --git a/src/app/modules/agent/intent_router_v2/retrieval_planning/evidence_policy_factory.py b/src/app/modules/agent/intent_router_v2/retrieval_planning/evidence_policy_factory.py deleted file mode 100644 index 100e5b3..0000000 --- a/src/app/modules/agent/intent_router_v2/retrieval_planning/evidence_policy_factory.py +++ /dev/null @@ -1,41 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.models import EvidencePolicy - - -class EvidencePolicyFactory: - def build( - self, - intent: str, - *, - sub_intent: str = "EXPLAIN", - negations: list[str] | None = None, - has_user_anchor: bool = True, - ) -> EvidencePolicy: - negations_set = set(negations or []) - if intent == "CODE_QA": - if sub_intent == "OPEN_FILE": - return EvidencePolicy(require_def=False, require_flow=False, require_spec=False, allow_answer_without_evidence=False) - if sub_intent == "EXPLAIN_LOCAL": - return EvidencePolicy(require_def=True, require_flow=False, require_spec=False, allow_answer_without_evidence=False) - if sub_intent == "TRACE_FLOW": - return EvidencePolicy(require_def=True, require_flow=True, require_spec=False, allow_answer_without_evidence=False) - if sub_intent == "ARCHITECTURE": - return EvidencePolicy(require_def=True, require_flow=True, require_spec=False, allow_answer_without_evidence=False) - if "tests" in negations_set and not has_user_anchor: - return EvidencePolicy(require_def=True, require_flow=False, require_spec=False, allow_answer_without_evidence=False) - return EvidencePolicy(require_def=True, require_flow=True, require_spec=False, allow_answer_without_evidence=False) - if intent in { - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - "DOCUMENTATION_DISCOVERY", - "DOCUMENTATION_NAVIGATION", - "OPENAPI_FROM_DOCUMENTATION", - }: - return EvidencePolicy(require_def=False, require_flow=False, require_spec=True, allow_answer_without_evidence=False) - if intent == "FALLBACK": - return EvidencePolicy(require_def=False, require_flow=False, require_spec=False, allow_answer_without_evidence=True) - if intent == "GENERATE_DOCS_FROM_CODE": - return EvidencePolicy(require_def=True, require_flow=False, require_spec=False, allow_answer_without_evidence=False) - return EvidencePolicy(require_def=False, require_flow=False, require_spec=False, allow_answer_without_evidence=True) diff --git a/src/app/modules/agent/intent_router_v2/retrieval_planning/layer_query_builder.py b/src/app/modules/agent/intent_router_v2/retrieval_planning/layer_query_builder.py deleted file mode 100644 index 911d493..0000000 --- a/src/app/modules/agent/intent_router_v2/retrieval_planning/layer_query_builder.py +++ /dev/null @@ -1,29 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.models import LayerQuery, RepoContext - - -class LayerQueryBuilder: - def build(self, intent: str, repo_context: RepoContext, *, domains: list[str], layers_map: dict[str, list[tuple[str, int]]]) -> list[LayerQuery]: - available = set(repo_context.available_layers or []) - result: list[LayerQuery] = [] - for layer_id, top_k in layers_map[intent]: - if not self._layer_matches_domains(layer_id, domains): - continue - if available and layer_id not in available: - continue - result.append(LayerQuery(layer_id=layer_id, top_k=top_k)) - if result: - return result - return [ - LayerQuery(layer_id=layer_id, top_k=top_k) - for layer_id, top_k in layers_map[intent] - if self._layer_matches_domains(layer_id, domains) - ] - - def _layer_matches_domains(self, layer_id: str, domains: list[str]) -> bool: - if domains == ["CODE"]: - return layer_id.startswith("C") - if domains == ["DOCS"]: - return layer_id.startswith("D") - return layer_id.startswith("C") or layer_id.startswith("D") diff --git a/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_constraints_factory.py b/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_constraints_factory.py deleted file mode 100644 index 14e2198..0000000 --- a/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_constraints_factory.py +++ /dev/null @@ -1,118 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.models import QueryAnchor, RetrievalConstraints, RetrievalProfile -from app.modules.agent.intent_router_v2.analysis.test_signals import has_test_focus, is_negative_test_request - - -class RetrievalConstraintsFactory: - _DEFAULT_DOCS_INCLUDE = ["docs/**", "README*", "**/*.md"] - _DEFAULT_DOCS_EXCLUDE = [".venv/**", "node_modules/**", "**/*.bin", "**/*.png", "**/*.jpg"] - _DEFAULT_EXCLUDE_TESTS = [ - "tests/**", - "**/test_*.py", - "**/*_test.py", - ] - _TEST_FILE_GLOBS = [ - "tests/**", - "**/test_*.py", - "**/*_test.py", - "**/conftest.py", - ] - - def build( - self, - *, - retrieval_profile: RetrievalProfile, - sub_intent: str, - raw_query: str, - anchors: list[QueryAnchor], - path_scope: list[str], - ) -> RetrievalConstraints: - if retrieval_profile in {"docs", "fallback"}: - return self._docs_constraints(sub_intent=sub_intent, path_scope=path_scope) - return self._code_constraints(sub_intent=sub_intent, raw_query=raw_query, anchors=anchors) - - def _code_constraints(self, *, sub_intent: str, raw_query: str, anchors: list[QueryAnchor]) -> RetrievalConstraints: - include_tests = sub_intent == "FIND_TESTS" or has_test_focus(raw_query) - exclude_tests = is_negative_test_request(raw_query) - if include_tests: - include_globs = ["src/**", *self._TEST_FILE_GLOBS] - prefer_globs = ["tests/**", "**/test_*.py", "**/*_test.py"] - exclude_globs: list[str] = [] - else: - include_globs = ["src/**"] - prefer_globs = [] - exclude_globs = list(self._DEFAULT_EXCLUDE_TESTS) - if exclude_tests: - include_globs = ["src/**"] - prefer_globs = [] - exclude_globs = list(self._DEFAULT_EXCLUDE_TESTS) - if sub_intent == "OPEN_FILE": - include_globs = self._open_file_globs(anchors) or include_globs - prefer_globs = [] - test_file_globs = list(self._TEST_FILE_GLOBS) if sub_intent == "FIND_TESTS" else [] - test_symbol_patterns = self._test_symbol_patterns(anchors) if sub_intent == "FIND_TESTS" else [] - return RetrievalConstraints( - include_globs=include_globs, - exclude_globs=exclude_globs, - prefer_globs=prefer_globs, - test_file_globs=test_file_globs, - test_symbol_patterns=test_symbol_patterns, - max_candidates=20, - fuzzy_symbol_search={ - "enabled": sub_intent != "OPEN_FILE", - "max_distance": 2, - "top_k": 5, - }, - ) - - def _docs_constraints(self, *, sub_intent: str, path_scope: list[str]) -> RetrievalConstraints: - include_globs = self._dedupe([*path_scope, *self._DEFAULT_DOCS_INCLUDE]) if path_scope else list(self._DEFAULT_DOCS_INCLUDE) - return RetrievalConstraints( - include_globs=include_globs, - exclude_globs=list(self._DEFAULT_DOCS_EXCLUDE), - prefer_globs=[], - test_file_globs=[], - test_symbol_patterns=[], - max_candidates=20, - fuzzy_symbol_search={ - "enabled": sub_intent != "OPEN_FILE", - "max_distance": 2, - "top_k": 5, - }, - ) - - def _open_file_globs(self, anchors: list[QueryAnchor]) -> list[str]: - for anchor in anchors: - if anchor.type != "FILE_PATH": - continue - value = anchor.value.strip() - if not value: - continue - return [value] - return [] - - def _test_symbol_patterns(self, anchors: list[QueryAnchor]) -> list[str]: - result: list[str] = [] - symbols = [anchor.value for anchor in anchors if anchor.type == "SYMBOL"] - for symbol in symbols[:3]: - snake = self._camel_to_snake(symbol) - for pattern in (f"test_{snake}", f"Test{symbol}", symbol): - if pattern and pattern not in result: - result.append(pattern) - return result - - def _camel_to_snake(self, value: str) -> str: - chars: list[str] = [] - for index, char in enumerate(value): - if index > 0 and char.isupper() and value[index - 1].isalnum(): - chars.append("_") - chars.append(char.lower()) - return "".join(chars).strip("_") - - def _dedupe(self, values: list[str]) -> list[str]: - result: list[str] = [] - for value in values: - if value and value not in result: - result.append(value) - return result diff --git a/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_filter_builder.py b/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_filter_builder.py deleted file mode 100644 index 7c36905..0000000 --- a/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_filter_builder.py +++ /dev/null @@ -1,120 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.models import ( - CodeRetrievalFilters, - ConversationState, - DocsRetrievalFilters, - HybridRetrievalFilters, - QueryAnchor, - RepoContext, -) -from app.modules.agent.intent_router_v2.analysis.test_signals import has_test_focus, is_negative_test_request, is_test_related_token - - -class RetrievalFilterBuilder: - def build( - self, - domains: list[str], - anchors: list[QueryAnchor], - repo_context: RepoContext, - *, - raw_query: str, - conversation_state: ConversationState | None, - conversation_mode: str, - sub_intent: str = "EXPLAIN", - ) -> CodeRetrievalFilters | DocsRetrievalFilters | HybridRetrievalFilters: - path_scope = self._path_scope( - anchors, - conversation_state=conversation_state, - conversation_mode=conversation_mode, - raw_query=raw_query, - sub_intent=sub_intent, - ) - if domains == ["DOCS"]: - return DocsRetrievalFilters( - path_scope=path_scope, - doc_kinds=self._doc_kinds(anchors, raw_query), - doc_language=[], - doc_type=self._doc_type(sub_intent), - ) - if domains == ["CODE"]: - return CodeRetrievalFilters( - test_policy=self._test_policy(raw_query, anchors, sub_intent=sub_intent), - path_scope=path_scope, - language=list(repo_context.languages), - ) - return HybridRetrievalFilters( - test_policy=self._test_policy(raw_query, anchors, sub_intent=sub_intent), - path_scope=path_scope, - language=list(repo_context.languages), - doc_kinds=self._doc_kinds(anchors, raw_query), - doc_language=[], - doc_type=self._doc_type(sub_intent), - ) - - def _test_policy(self, raw_query: str, anchors: list[QueryAnchor], *, sub_intent: str) -> str: - if sub_intent == "FIND_TESTS": - return "INCLUDE" - if is_negative_test_request(raw_query): - return "EXCLUDE" - if has_test_focus(raw_query): - return "INCLUDE" - has_test_keyterm = any(anchor.type == "KEY_TERM" and is_test_related_token(anchor.value) for anchor in anchors) - return "INCLUDE" if has_test_keyterm else "EXCLUDE" - - def _path_scope( - self, - anchors: list[QueryAnchor], - *, - conversation_state: ConversationState | None, - conversation_mode: str, - raw_query: str, - sub_intent: str, - ) -> list[str]: - values: list[str] = [] - has_user_file_anchor = False - file_values: list[str] = [] - for anchor in anchors: - if anchor.type != "FILE_PATH": - continue - if anchor.source == "user_text": - has_user_file_anchor = True - if anchor.value not in values: - values.append(anchor.value) - if self._looks_like_file_path(anchor.value) and anchor.value not in file_values: - file_values.append(anchor.value) - parent = anchor.value.rsplit("/", 1)[0] if "/" in anchor.value and self._looks_like_file_path(anchor.value) else "" - if parent and parent not in values: - values.append(parent) - if sub_intent in {"OPEN_FILE", "EXPLAIN_LOCAL"} and file_values and not self._is_explicit_directory_scope(raw_query): - return file_values[:6] - if has_user_file_anchor or conversation_mode != "CONTINUE": - return values[:6] - if values: - return values[:6] - inherited = list((conversation_state.active_path_scope if conversation_state else []) or []) - return inherited[:6] - - def _doc_kinds(self, anchors: list[QueryAnchor], raw_query: str) -> list[str]: - text = (raw_query or "").lower() - kinds: list[str] = [] - has_readme = "readme" in text or any( - anchor.type in {"DOC_REF", "FILE_PATH"} and anchor.value.lower().endswith("readme.md") - for anchor in anchors - ) - if has_readme: - kinds.append("README") - return kinds - - def _doc_type(self, sub_intent: str) -> str | None: - if sub_intent in {"API_METHOD_EXPLAIN", "OPENAPI_METHOD_GENERATE", "OPENAPI_FRAGMENT_GENERATE"}: - return "api_method" - return None - - def _looks_like_file_path(self, value: str) -> bool: - filename = value.rsplit("/", 1)[-1] - return "." in filename - - def _is_explicit_directory_scope(self, raw_query: str) -> bool: - text = (raw_query or "").lower() - return any(marker in text for marker in ("в папке", "в директории", "в каталоге")) diff --git a/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_spec_factory.py b/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_spec_factory.py deleted file mode 100644 index 8a55c41..0000000 --- a/src/app/modules/agent/intent_router_v2/retrieval_planning/retrieval_spec_factory.py +++ /dev/null @@ -1,227 +0,0 @@ -from __future__ import annotations - -from app.modules.rag.contracts.enums import RagLayer -from app.modules.agent.intent_router_v2.retrieval_planning.layer_query_builder import LayerQueryBuilder -from app.modules.agent.intent_router_v2.models import ConversationState, QueryAnchor, RepoContext, RetrievalSpec -from app.modules.agent.intent_router_v2.retrieval_planning.retrieval_filter_builder import RetrievalFilterBuilder - - -class RetrievalSpecFactory: - _LAYERS = { - "CODE_QA": [ - (RagLayer.CODE_SYMBOL_CATALOG, 8), - (RagLayer.CODE_SOURCE_CHUNKS, 8), - (RagLayer.CODE_SEMANTIC_ROLES, 8), - (RagLayer.CODE_DEPENDENCY_GRAPH, 6), - (RagLayer.CODE_ENTRYPOINTS, 6), - ], - "DOCUMENTATION_EXPLAIN": [ - (RagLayer.DOCS_DOCUMENT_CATALOG, 6), - (RagLayer.DOCS_FACT_INDEX, 8), - (RagLayer.DOCS_RELATION_GRAPH, 6), - ], - "OPENAPI_GENERATION": [ - (RagLayer.DOCS_DOCUMENT_CATALOG, 8), - (RagLayer.DOCS_FACT_INDEX, 8), - (RagLayer.DOCS_DOC_CHUNKS, 6), - ], - "GENERAL_QA": [ - (RagLayer.DOCS_DOCUMENT_CATALOG, 4), - (RagLayer.DOCS_DOC_CHUNKS, 4), - ], - "GENERATE_DOCS_FROM_CODE": [ - (RagLayer.CODE_SYMBOL_CATALOG, 12), - (RagLayer.CODE_DEPENDENCY_GRAPH, 8), - (RagLayer.CODE_SOURCE_CHUNKS, 12), - (RagLayer.CODE_ENTRYPOINTS, 6), - ], - "FALLBACK": [ - (RagLayer.DOCS_DOCUMENT_CATALOG, 4), - (RagLayer.DOCS_DOC_CHUNKS, 6), - ], - } - _DOMAINS = { - "CODE_QA": ["CODE"], - "DOCUMENTATION_EXPLAIN": ["DOCS"], - "OPENAPI_GENERATION": ["DOCS"], - "GENERAL_QA": ["DOCS"], - "GENERATE_DOCS_FROM_CODE": ["CODE"], - "FALLBACK": ["DOCS"], - } - _RERANK = { - "CODE_QA": "code", - "DOCUMENTATION_EXPLAIN": "docs", - "OPENAPI_GENERATION": "docs", - "GENERAL_QA": "fallback", - "GENERATE_DOCS_FROM_CODE": "generate", - "FALLBACK": "fallback", - } - _OPEN_FILE_LAYERS = [ - (RagLayer.CODE_SOURCE_CHUNKS, 12), - ] - _FIND_ENTRYPOINTS_LAYERS = [ - (RagLayer.CODE_ENTRYPOINTS, 12), - (RagLayer.CODE_SOURCE_CHUNKS, 6), - ] - _EXPLAIN_LOCAL_LAYERS = [ - (RagLayer.CODE_SYMBOL_CATALOG, 8), - (RagLayer.CODE_SOURCE_CHUNKS, 12), - (RagLayer.CODE_DEPENDENCY_GRAPH, 4), - ] - _FIND_TESTS_LAYERS = [ - (RagLayer.CODE_SYMBOL_CATALOG, 8), - (RagLayer.CODE_DEPENDENCY_GRAPH, 6), - (RagLayer.CODE_SOURCE_CHUNKS, 10), - ] - _TRACE_FLOW_LAYERS = [ - (RagLayer.CODE_SYMBOL_CATALOG, 8), - (RagLayer.CODE_SOURCE_CHUNKS, 8), - (RagLayer.CODE_SEMANTIC_ROLES, 8), - (RagLayer.CODE_DEPENDENCY_GRAPH, 10), - (RagLayer.CODE_ENTRYPOINTS, 10), - ] - _ARCHITECTURE_LAYERS = [ - (RagLayer.CODE_SEMANTIC_ROLES, 12), - (RagLayer.CODE_ENTRYPOINTS, 8), - (RagLayer.CODE_DEPENDENCY_GRAPH, 8), - (RagLayer.CODE_SYMBOL_CATALOG, 6), - (RagLayer.CODE_SOURCE_CHUNKS, 4), - ] - _DOCS_SCOPED_LAYERS = [(RagLayer.DOCS_DOC_CHUNKS, 8)] - _DOCS_SYSTEM_FLOW_LAYERS = [ - (RagLayer.DOCS_WORKFLOW_INDEX, 8), - (RagLayer.DOCS_RELATION_GRAPH, 8), - (RagLayer.DOCS_DOCUMENT_CATALOG, 4), - (RagLayer.DOCS_DOC_CHUNKS, 4), - ] - _DOCS_COMPONENT_LAYERS = [ - (RagLayer.DOCS_FACT_INDEX, 8), - (RagLayer.DOCS_RELATION_GRAPH, 8), - (RagLayer.DOCS_DOCUMENT_CATALOG, 4), - (RagLayer.DOCS_DOC_CHUNKS, 4), - ] - _DOCS_API_METHOD_LAYERS = [ - (RagLayer.DOCS_FACT_INDEX, 8), - (RagLayer.DOCS_WORKFLOW_INDEX, 8), - (RagLayer.DOCS_DOCUMENT_CATALOG, 4), - (RagLayer.DOCS_DOC_CHUNKS, 4), - ] - _DOCS_ENTITY_LAYERS = [ - (RagLayer.DOCS_ENTITY_CATALOG, 8), - (RagLayer.DOCS_RELATION_GRAPH, 8), - (RagLayer.DOCS_DOCUMENT_CATALOG, 4), - (RagLayer.DOCS_DOC_CHUNKS, 4), - ] - _DOCS_RELATED_LAYERS = [ - (RagLayer.DOCS_RELATION_GRAPH, 8), - (RagLayer.DOCS_DOCUMENT_CATALOG, 4), - (RagLayer.DOCS_DOC_CHUNKS, 3), - ] - _DOCS_OPENAPI_LAYERS = [ - (RagLayer.DOCS_DOCUMENT_CATALOG, 8), - (RagLayer.DOCS_FACT_INDEX, 8), - (RagLayer.DOCS_DOC_CHUNKS, 6), - ] - - def __init__( - self, - layer_builder: LayerQueryBuilder | None = None, - filter_builder: RetrievalFilterBuilder | None = None, - ) -> None: - self._layer_builder = layer_builder or LayerQueryBuilder() - self._filter_builder = filter_builder or RetrievalFilterBuilder() - - def build( - self, - intent: str, - anchors: list[QueryAnchor], - repo_context: RepoContext, - *, - raw_query: str = "", - conversation_state: ConversationState | None = None, - conversation_mode: str = "START", - sub_intent: str = "EXPLAIN", - ) -> RetrievalSpec: - domains = self._domains(intent, repo_context) - layers_map = self._with_sub_intent_layers(intent, sub_intent, anchors) - layer_queries = self._layer_builder.build(intent, repo_context, domains=domains, layers_map=layers_map) - filters = self._filter_builder.build( - domains, - anchors, - repo_context, - raw_query=raw_query, - conversation_state=conversation_state, - conversation_mode=conversation_mode, - sub_intent=sub_intent, - ) - if intent == "DOCUMENTATION_EXPLAIN" and list(getattr(filters, "path_scope", []) or []): - scoped_map = dict(self._LAYERS) - scoped_map[intent] = list(self._DOCS_SCOPED_LAYERS) - layer_queries = self._layer_builder.build(intent, repo_context, domains=domains, layers_map=scoped_map) - return RetrievalSpec( - domains=domains, - layer_queries=layer_queries, - filters=filters, - rerank_profile=self._RERANK[intent], - ) - - def _domains(self, intent: str, repo_context: RepoContext) -> list[str]: - available = set(repo_context.available_domains or ["CODE", "DOCS"]) - result = [domain for domain in self._DOMAINS[intent] if domain in available] - return result or list(self._DOMAINS[intent]) - - def _with_sub_intent_layers( - self, - intent: str, - sub_intent: str, - anchors: list[QueryAnchor], - ) -> dict[str, list[tuple[str, int]]]: - if intent in {"DOCUMENTATION_EXPLAIN", "OPENAPI_GENERATION", "GENERAL_QA"}: - return self._with_docs_sub_intent_layers(intent, sub_intent) - if intent != "CODE_QA": - return self._LAYERS - layers_map = dict(self._LAYERS) - if sub_intent == "OPEN_FILE": - layers_map["CODE_QA"] = list(self._OPEN_FILE_LAYERS) - elif sub_intent == "FIND_ENTRYPOINTS": - layers_map["CODE_QA"] = list(self._FIND_ENTRYPOINTS_LAYERS) - elif sub_intent == "FIND_TESTS": - layers_map["CODE_QA"] = list(self._FIND_TESTS_LAYERS) - elif sub_intent == "TRACE_FLOW": - layers_map["CODE_QA"] = list(self._TRACE_FLOW_LAYERS) - elif sub_intent == "ARCHITECTURE": - layers_map["CODE_QA"] = list(self._ARCHITECTURE_LAYERS) - elif sub_intent == "EXPLAIN_LOCAL": - layers_map["CODE_QA"] = list(self._EXPLAIN_LOCAL_LAYERS) - elif sub_intent != "EXPLAIN" and not self._needs_entrypoints(anchors): - layers_map["CODE_QA"] = [ - (RagLayer.CODE_SYMBOL_CATALOG, 8), - (RagLayer.CODE_SOURCE_CHUNKS, 8), - (RagLayer.CODE_DEPENDENCY_GRAPH, 6), - ] - return layers_map - - def _with_docs_sub_intent_layers(self, intent: str, sub_intent: str) -> dict[str, list[tuple[str, int]]]: - layers_map = dict(self._LAYERS) - if intent == "DOCUMENTATION_EXPLAIN": - if sub_intent == "SYSTEM_FLOW_EXPLAIN": - layers_map[intent] = list(self._DOCS_SYSTEM_FLOW_LAYERS) - elif sub_intent == "API_METHOD_EXPLAIN": - layers_map[intent] = list(self._DOCS_API_METHOD_LAYERS) - elif sub_intent == "ENTITY_EXPLAIN": - layers_map[intent] = list(self._DOCS_ENTITY_LAYERS) - elif sub_intent == "RELATED_DOCS_EXPLAIN": - layers_map[intent] = list(self._DOCS_RELATED_LAYERS) - else: - layers_map[intent] = list(self._DOCS_COMPONENT_LAYERS) - return layers_map - if intent == "GENERAL_QA": - layers_map[intent] = list(self._LAYERS["GENERAL_QA"]) - return layers_map - layers_map[intent] = list(self._DOCS_OPENAPI_LAYERS) - return layers_map - - def _needs_entrypoints(self, anchors: list[QueryAnchor]) -> bool: - values = " ".join(anchor.value.lower() for anchor in anchors if anchor.type in {"KEY_TERM", "SYMBOL"}) - markers = ("entrypoint", "endpoint", "вызыва", "поток", "flow", "запуска") - return any(marker in values for marker in markers) diff --git a/src/app/modules/agent/intent_router_v2/router.py b/src/app/modules/agent/intent_router_v2/router.py deleted file mode 100644 index 2bca83d..0000000 --- a/src/app/modules/agent/intent_router_v2/router.py +++ /dev/null @@ -1,186 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2.analysis.ambiguity_detector import DocsAmbiguityDetector -from app.modules.agent.intent_router_v2.intent.classifier import IntentClassifierV2 -from app.modules.agent.intent_router_v2.analysis.docs_query_signals import DocsQuerySignals -from app.modules.agent.intent_router_v2.intent.llm_disambiguator import DocsLlmDisambiguator -from app.modules.agent.intent_router_v2.intent.conversation_policy import ConversationPolicy -from app.modules.agent.intent_router_v2.retrieval_planning.evidence_policy_factory import EvidencePolicyFactory -from app.modules.agent.intent_router_v2.intent.graph_id_resolver import GraphIdResolver -from app.modules.agent.intent_router_v2.logger import IntentRouterLogger -from app.modules.agent.intent_router_v2.models import ConversationState, IntentRouterResult, RepoContext, SymbolResolution -from app.modules.agent.intent_router_v2.analysis.query_plan_builder import QueryPlanBuilder -from app.modules.agent.intent_router_v2.retrieval_planning.retrieval_constraints_factory import RetrievalConstraintsFactory -from app.modules.agent.intent_router_v2.retrieval_planning.retrieval_spec_factory import RetrievalSpecFactory - - -class IntentRouterV2: - def __init__( - self, - classifier: IntentClassifierV2 | None = None, - conversation_policy: ConversationPolicy | None = None, - query_plan_builder: QueryPlanBuilder | None = None, - retrieval_factory: RetrievalSpecFactory | None = None, - constraints_factory: RetrievalConstraintsFactory | None = None, - evidence_factory: EvidencePolicyFactory | None = None, - graph_resolver: GraphIdResolver | None = None, - logger: IntentRouterLogger | None = None, - ambiguity_detector: DocsAmbiguityDetector | None = None, - llm_disambiguator: DocsLlmDisambiguator | None = None, - enable_llm_disambiguation: bool = False, - ) -> None: - self._classifier = classifier or IntentClassifierV2() - self._conversation_policy = conversation_policy or ConversationPolicy() - self._query_plan_builder = query_plan_builder or QueryPlanBuilder() - self._retrieval_factory = retrieval_factory or RetrievalSpecFactory() - self._constraints_factory = constraints_factory or RetrievalConstraintsFactory() - self._evidence_factory = evidence_factory or EvidencePolicyFactory() - self._graph_resolver = graph_resolver or GraphIdResolver() - self._logger = logger or IntentRouterLogger() - self._ambiguity_detector = ambiguity_detector or DocsAmbiguityDetector() - self._llm_disambiguator = llm_disambiguator - self._enable_llm_disambiguation = enable_llm_disambiguation - self._docs_signals = DocsQuerySignals() - - def route( - self, - user_query: str, - conversation_state: ConversationState | None = None, - repo_context: RepoContext | None = None, - ) -> IntentRouterResult: - state = conversation_state or ConversationState() - context = repo_context or RepoContext() - self._logger.log_request(user_query, state, context) - decision = self._classifier.classify(user_query, state) - intent, conversation_mode = self._conversation_policy.resolve(decision, user_query, state) - retrieval_profile = self._resolve_retrieval_profile(intent) - query_plan = self._query_plan_builder.build( - user_query, - state, - continue_mode=conversation_mode in {"CONTINUE", "FOLLOWUP_LIKELY"}, - conversation_mode=conversation_mode, - intent=intent, - ) - ambiguity = self._ambiguity_detector.detect(user_query, intent=intent, sub_intent=query_plan.sub_intent) - query_plan, intent, routing_mode, matched_source, llm_info, routing_reason = self._resolve_final_routing( - user_query=user_query, - query_plan=query_plan, - intent=intent, - ambiguity=ambiguity, - routing_reason=str(decision.reason or ""), - ) - retrieval_spec = self._retrieval_factory.build( - intent, - query_plan.anchors, - context, - raw_query=query_plan.raw, - conversation_state=state, - conversation_mode=conversation_mode, - sub_intent=query_plan.sub_intent, - ) - path_scope = list(getattr(retrieval_spec.filters, "path_scope", []) or []) - matched_anchor_type, matched_anchor_value = self._docs_signals.detect_anchor(user_query) - result = IntentRouterResult( - intent=intent, - retrieval_profile=retrieval_profile, - graph_id=self._graph_resolver.resolve(intent), - conversation_mode=conversation_mode, - query_plan=query_plan, - retrieval_spec=retrieval_spec, - retrieval_constraints=self._constraints_factory.build( - retrieval_profile=retrieval_profile, - sub_intent=query_plan.sub_intent, - raw_query=query_plan.raw, - anchors=query_plan.anchors, - path_scope=path_scope, - ), - symbol_resolution=self._initial_symbol_resolution( - retrieval_profile, - query_plan.sub_intent, - query_plan.symbol_candidates, - ), - evidence_policy=self._evidence_factory.build( - intent, - sub_intent=query_plan.sub_intent, - negations=query_plan.negations, - has_user_anchor=any(anchor.source == "user_text" for anchor in query_plan.anchors), - ), - matched_anchor_type=matched_anchor_type, # type: ignore[arg-type] - matched_anchor_value=matched_anchor_value, - matched_intent_source=matched_source, - routing_reason=routing_reason, - routing_mode=routing_mode, - is_ambiguous=bool(ambiguity.get("is_ambiguous")), - ambiguity_reason=str(ambiguity.get("ambiguity_reason") or ""), - deterministic_candidates=list(ambiguity.get("deterministic_candidates") or []), - deterministic_selected_sub_intent=str(ambiguity.get("deterministic_primary_candidate") or query_plan.sub_intent), - llm_router_used=bool(llm_info.get("used")), - llm_router_selected_sub_intent=str(llm_info.get("sub_intent") or ""), - llm_router_reason=str(llm_info.get("reason") or ""), - llm_router_confidence=str(llm_info.get("confidence") or ""), - llm_router_error=str(llm_info.get("error") or ""), - ) - self._logger.log_result(result) - return result - - def _initial_symbol_resolution( - self, - retrieval_profile: str, - sub_intent: str, - symbol_candidates: list[str], - ) -> SymbolResolution: - if retrieval_profile in {"docs", "fallback"}: - return SymbolResolution(status="not_requested") - if sub_intent == "OPEN_FILE": - return SymbolResolution(status="not_requested") - if not symbol_candidates: - return SymbolResolution(status="not_requested") - return SymbolResolution( - status="pending", - resolved_symbol=None, - alternatives=list(symbol_candidates), - confidence=0.0, - ) - - def _resolve_retrieval_profile(self, intent: str) -> str: - if intent in {"DOCUMENTATION_EXPLAIN", "OPENAPI_GENERATION", "GENERAL_QA"}: - return "docs" - if intent == "FALLBACK": - return "fallback" - return "code" - - def _resolve_final_routing( - self, - *, - user_query: str, - query_plan, - intent: str, - ambiguity: dict[str, object], - routing_reason: str, - ): - if not bool(ambiguity.get("is_ambiguous")): - source = "llm" if not str(routing_reason).startswith("deterministic_") else "deterministic" - return query_plan, intent, "deterministic", source, {"used": False}, routing_reason - if not self._enable_llm_disambiguation or self._llm_disambiguator is None: - return query_plan, intent, "deterministic_fallback", "deterministic", {"used": False}, routing_reason - payload = { - "query": user_query, - "normalized_query": query_plan.normalized, - "deterministic_primary_candidate": str(ambiguity.get("deterministic_primary_candidate") or query_plan.sub_intent), - "deterministic_candidates": list(ambiguity.get("deterministic_candidates") or []), - "ambiguity_reason": str(ambiguity.get("ambiguity_reason") or ""), - "matched_anchor_type": self._docs_signals.detect_anchor(user_query)[0], - "query_entity_candidates": self._docs_signals.query_entity_candidates(user_query), - "query_anchor_candidates": self._docs_signals.query_anchor_candidates(user_query), - } - try: - llm_choice = self._llm_disambiguator.choose(payload) - except Exception as exc: - return query_plan, intent, "deterministic_fallback", "deterministic", {"used": False, "error": str(exc)}, routing_reason - if llm_choice is None: - return query_plan, intent, "deterministic_fallback", "deterministic", {"used": False, "error": "invalid_llm_output"}, routing_reason - final_sub_intent = str(llm_choice.get("sub_intent") or query_plan.sub_intent) - final_intent = "GENERAL_QA" if final_sub_intent == "GENERIC_QA" else intent - final_intent = "DOCUMENTATION_EXPLAIN" if final_sub_intent in {"SYSTEM_FLOW_EXPLAIN", "COMPONENT_EXPLAIN", "API_METHOD_EXPLAIN", "ENTITY_EXPLAIN", "RELATED_DOCS_EXPLAIN"} else final_intent - final_query_plan = query_plan.model_copy(update={"sub_intent": final_sub_intent}) - return final_query_plan, final_intent, "llm_disambiguation", "llm", {"used": True, **llm_choice}, f"llm_disambiguation:{llm_choice.get('reason') or 'override'}" diff --git a/src/app/modules/agent/llm/__init__.py b/src/app/modules/agent/llm/__init__.py deleted file mode 100644 index ba06118..0000000 --- a/src/app/modules/agent/llm/__init__.py +++ /dev/null @@ -1,4 +0,0 @@ -from app.modules.agent.llm.service import AgentLlmService -from app.modules.agent.llm.prompt_loader import PromptLoader - -__all__ = ["AgentLlmService", "PromptLoader"] diff --git a/src/app/modules/agent/llm/prompt_loader.py b/src/app/modules/agent/llm/prompt_loader.py deleted file mode 100644 index e432c78..0000000 --- a/src/app/modules/agent/llm/prompt_loader.py +++ /dev/null @@ -1,27 +0,0 @@ -from pathlib import Path -import os - -import yaml - - -class PromptLoader: - def __init__(self, prompts_path: Path | None = None) -> None: - base = prompts_path or Path(__file__).resolve().parent / "prompts.yml" - env_override = os.getenv("AGENT_PROMPTS_DIR", "").strip() - raw_path = Path(env_override) if env_override else base - self._path = raw_path / "prompts.yml" if raw_path.is_dir() else raw_path - self._prompts = self._load_prompts() - - def load(self, name: str) -> str: - return str(self._prompts.get(name, "") or "").strip() - - def _load_prompts(self) -> dict[str, str]: - if not self._path.is_file(): - return {} - payload = yaml.safe_load(self._path.read_text(encoding="utf-8")) or {} - if not isinstance(payload, dict): - return {} - prompts = payload.get("prompts", payload) - if not isinstance(prompts, dict): - return {} - return {str(key): str(value or "") for key, value in prompts.items()} diff --git a/src/app/modules/agent/llm/prompts.yml b/src/app/modules/agent/llm/prompts.yml deleted file mode 100644 index 48c7bf1..0000000 --- a/src/app/modules/agent/llm/prompts.yml +++ /dev/null @@ -1,386 +0,0 @@ -prompts: - code_explain_answer_v2: | - Объяснение кода осуществляется только с использованием предоставленного ExplainPack. - - Правила: - - Сначала используйте доказательства. - - Каждый ключевой шаг в процессе должен содержать один или несколько идентификаторов доказательств в квадратных скобках, например, [entrypoint_1] или [excerpt_3]. - - Не придумывайте символы, файлы, маршруты или фрагменты кода, отсутствующие в пакете. - - Если доказательства неполные, укажите это явно. - - В качестве якорей используйте выбранные точки входа и пути трассировки. - - Верните Markdown со следующей структурой: - 1. Краткое описание - 2. Пошаговый процесс - 3. Данные и побочные эффекты - 4. Ошибки и граничные случаи - 5. Указатели - - Указатели должны представлять собой короткий маркированный список, сопоставляющий идентификаторы доказательств с местоположениями файлов. - code_qa_architecture_answer: | - Ты инженер, который объясняет устройство подсистемы только по наблюдаемым компонентам и связям из кода. - - В payload есть `answer_contract`, `must_mention_components`, `must_mention_relations`, `must_mention_relation_summaries`, `must_use_relation_verbs`. Это обязательный каркас: ответ должен перечислять компоненты уровня класса/модуля и связи между ними с глаголами (создаёт, вызывает, импортирует, читает, записывает, наследует). Учитывай `fact_gaps`. - - Отвечай только по коду из контекста. Пиши естественным языком, без лишних markdown-секций. - Строй ответ вокруг компонентов из `must_mention_components` и связей из `must_mention_relations` / `must_mention_relation_summaries`. Каждую связь формулируй с relation verb из `must_use_relation_verbs`. Методы и функции упоминай только как обоснование связи (например, "A вызывает B через метод X"), а не как основные "компоненты" списка. - Запрещено использовать в ответе raw retrieval labels: dataflow_slice, execution_trace, trace_path. Запрещено подменять архитектуру перечислением одних только методов без компонентов и связей. Запрещены абстрактные формулы без опоры на payload: "главный компонент", "управляет потоками данных", "этап пайплайна". - Не используй semantic_hints как primary explanation. Если связей в payload нет (fact_gaps), так и скажи — не додумывай связи. Не расширяй архитектуру за пределы извлечённого контекста. - code_qa_degraded_answer: | - Ты формируешь осторожный деградированный ответ. - Нужно честно описать, что удалось подтвердить, а чего не хватает. - Не выдавай предположения за факты и не заполняй пробелы догадками. - code_qa_explain_answer: | - Ты senior Python-инженер и code reviewer, который объясняет устройство кода без домысливания. - - В payload есть блок `answer_contract` и списки must_mention_* — это обязательный каркас ответа. Если списки непусты, ты обязан использовать из них конкретные имена (методы, вызовы, зависимости, поля), а не подменять их общими фразами. Учитывай `fact_gaps`: если там указаны пробелы в данных, явно скажи об этом и не додумывай. - - Отвечай только по коду из контекста. Пиши естественным инженерным языком, без лишних markdown-секций. - Начни с идентификации сущности и её расположения. Затем обязательно опирайся на: `must_mention_methods`, `must_mention_calls`, `must_mention_dependencies`, `must_mention_fields`, `must_mention_constructor_args`, `must_mention_files`. Каждый непустой список должен быть отражён в ответе конкретными именами из списка — хотя бы часть. Не заменяй их формулировками вроде "принимает ряд аргументов", "имеет responsibilities", "регистрирует основные службы", "используется в службах". - Если в fact_gaps указано, что методы или вызовы не подтверждены, прямо скажи об этом и не строй объяснение на догадках. - Запрещено использовать semantic_hints как основной каркас ответа; только concrete code edges (C0/C1/C2). Избегай расплывчатых фраз: "ряд аргументов", "ключевой компонент", "играет роль", "представляет собой" без конкретики. - Если сущность не найдена или evidence слабый — скажи об этом и остановись. Не используй обязательные секции и подзаголовки. - code_qa_explain_local_answer: | - Ты инженер, который объясняет локальный фрагмент кода без лишней теории и без перехода на уровень всей архитектуры. - - Отвечай только по коду и структуре проекта, которые есть в контексте. - Пиши естественным инженерным языком, без искусственных markdown-секций и без повторов одной и той же мысли. - Если ответ можно дать в 1-3 фразах, не раздувай его. - Упоминай файлы, классы, функции, методы и связи только если они реально присутствуют в извлечённых данных. - Каждое содержательное утверждение по возможности привязывай к конкретному наблюдаемому имени или факту из контекста: пути файла, имени класса, функции, метода, аргумента, поля, route path, вызова или связи. - Если конкретные имена, параметры, вызовы или связи не видны, прямо скажи, чего именно не видно, вместо общих формулировок. - Не вводи новые сущности, зависимости или сценарии, которых нет в контексте. - Явно различай подтверждённые факты и осторожные выводы по косвенным признакам. - Если данных мало, честно скажи об этом вместо общего обзора. - Не используй жирные заголовки блоков, если пользователь их не просил. - Строго соблюдай контракт sub-intent и не подменяй локальный ответ архитектурным обзором. - Избегай расплывчатых и пустых формулировок вроде: "различные аргументы", "ряд аргументов", "различные подпакеты", "основные службы", "ключевой компонент", "играет роль", "представляет собой", если после них нет конкретики. - Не добавляй очевидные метафразы о том, что ответ основан на контексте или на видимом фрагменте, если это ничего не добавляет по сути. - Если сущность не найдена, остановись на факте not_found и не объясняй её предполагаемое назначение по одному только названию. - Не выводи пустые разделы, пустые списки и формулировки вида "кандидатов нет", если это не помогает ответу. - - Дай локальное объяснение по конкретному файлу, символу или короткому участку кода. - Сконцентрируйся на том, что делает этот участок, какие входы и выходы видны и какие ближайшие вызовы или зависимости заметны рядом. - Если виден только фрагмент, ограничь вывод тем, что прямо видно в этом фрагменте. - Не компенсируй нехватку локального контекста общими архитектурными фразами. - Не расписывай всю архитектуру проекта и не используй секции без необходимости. - - code_qa_find_entrypoints_answer: | - Ты инженер, который находит подтверждённые точки входа и отдельно помечает только возможные кандидаты. - - Отвечай только по коду и структуре проекта, которые есть в контексте. - Пиши естественным инженерным языком, без искусственных markdown-секций и без повторов одной и той же мысли. - Если ответ можно дать в 1-3 фразах, не раздувай его. - Упоминай файлы, классы, функции, методы и связи только если они реально присутствуют в извлечённых данных. - Каждое содержательное утверждение по возможности привязывай к конкретному наблюдаемому имени или факту из контекста: пути файла, имени класса, функции, метода, аргумента, поля, route path, вызова или связи. - Если конкретные имена, параметры, вызовы или связи не видны, прямо скажи, чего именно не видно, вместо общих формулировок. - Не вводи новые сущности, зависимости или сценарии, которых нет в контексте. - Явно различай подтверждённые факты и осторожные выводы по косвенным признакам. - Если данных мало, честно скажи об этом вместо общего обзора. - Не используй жирные заголовки блоков, если пользователь их не просил. - Строго соблюдай контракт sub-intent и не подменяй локальный ответ архитектурным обзором. - Избегай расплывчатых и пустых формулировок вроде: "различные аргументы", "ряд аргументов", "различные подпакеты", "основные службы", "ключевой компонент", "играет роль", "представляет собой", если после них нет конкретики. - Не добавляй очевидные метафразы о том, что ответ основан на контексте или на видимом фрагменте, если это ничего не добавляет по сути. - Если сущность не найдена, остановись на факте not_found и не объясняй её предполагаемое назначение по одному только названию. - Не выводи пустые разделы, пустые списки и формулировки вида "кандидатов нет", если это не помогает ответу. - - Найди точки входа, обработчики запуска или важные entrypoints. - Для подтверждённых HTTP route сначала называй их в прикладном виде: HTTP method и route path, например `GET /health`. - Затем коротко добавляй, где route объявлен и какой handler, функция, метод или контекст его обслуживает, если это видно. - Если во входе есть `required_entrypoints`, каждый такой route должен быть явно назван в ответе в виде `METHOD /path`. - Если во входе есть `confirmed_entrypoints` с `query_match=true`, не пиши, что route не найден, пока не перечислишь эти совпавшие подтверждённые route. - Подтверждённые entrypoints перечисляй первыми. - Кандидатов без явного route marker упоминай только если они действительно полезны, и явно помечай как кандидатов. - Не своди ответ к обсуждению декораторов вроде `@app.get`; пользователю важнее method, path и контекст. - Не используй искусственные секции, если ответ можно дать компактным списком или коротким абзацем. - Если кандидатов нет, не создавай отдельную строку или блок про их отсутствие. - Не заменяй `GET /health` абстрактной формулой вроде "route для health-check"; сначала всегда пиши method и path. - code_qa_find_tests_answer: | - Ты инженер, который ищет тестовое покрытие и различает прямые и косвенные тесты. - - Отвечай только по коду и структуре проекта, которые есть в контексте. - Пиши естественным инженерным языком, без искусственных markdown-секций и без повторов одной и той же мысли. - Если ответ можно дать в 1-3 фразах, не раздувай его. - Упоминай файлы, классы, функции, методы и связи только если они реально присутствуют в извлечённых данных. - Каждое содержательное утверждение по возможности привязывай к конкретному наблюдаемому имени или факту из контекста: пути файла, имени класса, функции, метода, аргумента, поля, route path, вызова или связи. - Если конкретные имена, параметры, вызовы или связи не видны, прямо скажи, чего именно не видно, вместо общих формулировок. - Не вводи новые сущности, зависимости или сценарии, которых нет в контексте. - Явно различай подтверждённые факты и осторожные выводы по косвенным признакам. - Если данных мало, честно скажи об этом вместо общего обзора. - Не используй жирные заголовки блоков, если пользователь их не просил. - Строго соблюдай контракт sub-intent и не подменяй локальный ответ архитектурным обзором. - Избегай расплывчатых и пустых формулировок вроде: "различные аргументы", "ряд аргументов", "различные подпакеты", "основные службы", "ключевой компонент", "играет роль", "представляет собой", если после них нет конкретики. - Не добавляй очевидные метафразы о том, что ответ основан на контексте или на видимом фрагменте, если это ничего не добавляет по сути. - Если сущность не найдена, остановись на факте not_found и не объясняй её предполагаемое назначение по одному только названию. - Не выводи пустые разделы, пустые списки и формулировки вида "кандидатов нет", если это не помогает ответу. - - Найди связанные тесты и ответь, где они расположены. - Сначала назови прямые тесты, только если связь с сущностью подтверждается именем, импортом, вызовом или проверяемым поведением. - Если прямых тестов нет, прямо скажи это и только потом упомяни ближайшие косвенные тесты, если они есть. - Коротко поясни, что именно проверяется. - Не выдавай косвенные совпадения за подтверждённое покрытие и не используй отчётные секции без нужды. - Если косвенных тестов тоже нет, не добавляй отдельный пустой блок про их отсутствие. - code_qa_general_answer: | - Ты senior Python-инженер, который даёт обзорный ответ по подсистеме или проекту, но остаётся строго привязанным к коду из контекста. - - Отвечай только по коду и структуре проекта, которые есть в контексте. - Пиши естественным инженерным языком, без искусственных markdown-секций и без повторов одной и той же мысли. - Если ответ можно дать в 1-3 фразах, не раздувай его. - Упоминай файлы, классы, функции, методы и связи только если они реально присутствуют в извлечённых данных. - Каждое содержательное утверждение по возможности привязывай к конкретному наблюдаемому имени или факту из контекста: пути файла, имени класса, функции, метода, аргумента, поля, route path, вызова или связи. - Если конкретные имена, параметры, вызовы или связи не видны, прямо скажи, чего именно не видно, вместо общих формулировок. - Не вводи новые сущности, зависимости или сценарии, которых нет в контексте. - Явно различай подтверждённые факты и осторожные выводы по косвенным признакам. - Если данных мало, честно скажи об этом вместо общего обзора. - Не используй жирные заголовки блоков, если пользователь их не просил. - Строго соблюдай контракт sub-intent и не подменяй локальный ответ архитектурным обзором. - Избегай расплывчатых и пустых формулировок вроде: "различные аргументы", "ряд аргументов", "различные подпакеты", "основные службы", "ключевой компонент", "играет роль", "представляет собой", если после них нет конкретики. - Не добавляй очевидные метафразы о том, что ответ основан на контексте или на видимом фрагменте, если это ничего не добавляет по сути. - Если сущность не найдена, остановись на факте not_found и не объясняй её предполагаемое назначение по одному только названию. - Не выводи пустые разделы, пустые списки и формулировки вида "кандидатов нет", если это не помогает ответу. - - Дай обзорный ответ по вопросу пользователя о коде, подсистеме или сценарии работы. - Сначала скажи, что можно уверенно подтвердить по коду, затем коротко укажи, какие файлы, классы, функции или route это подтверждают. - Если данных недостаточно, прямо скажи, чего именно не хватает. - Не подменяй обзор общими рассуждениями о типичной архитектуре таких систем. - Не используй секции без необходимости. - Не заполняй пробелы общими словами вроде "несколько модулей", "различные компоненты" или "ряд зависимостей", если конкретные имена не видны. - code_qa_open_file_answer: | - Ты технический ассистент, который помогает открыть конкретный файл и показать, что в нём реально видно. - - Отвечай только по коду и структуре проекта, которые есть в контексте. - Пиши естественным инженерным языком, без искусственных markdown-секций и без повторов одной и той же мысли. - Если ответ можно дать в 1-3 фразах, не раздувай его. - Упоминай файлы, классы, функции, методы и связи только если они реально присутствуют в извлечённых данных. - Каждое содержательное утверждение по возможности привязывай к конкретному наблюдаемому имени или факту из контекста: пути файла, имени класса, функции, метода, аргумента, поля, route path, вызова или связи. - Если конкретные имена, параметры, вызовы или связи не видны, прямо скажи, чего именно не видно, вместо общих формулировок. - Не вводи новые сущности, зависимости или сценарии, которых нет в контексте. - Явно различай подтверждённые факты и осторожные выводы по косвенным признакам. - Если данных мало, честно скажи об этом вместо общего обзора. - Не используй жирные заголовки блоков, если пользователь их не просил. - Строго соблюдай контракт sub-intent и не подменяй локальный ответ архитектурным обзором. - Избегай расплывчатых и пустых формулировок вроде: "различные аргументы", "ряд аргументов", "различные подпакеты", "основные службы", "ключевой компонент", "играет роль", "представляет собой", если после них нет конкретики. - Не добавляй очевидные метафразы о том, что ответ основан на контексте или на видимом фрагменте, если это ничего не добавляет по сути. - Если сущность не найдена, остановись на факте not_found и не объясняй её предполагаемое назначение по одному только названию. - Не выводи пустые разделы, пустые списки и формулировки вида "кандидатов нет", если это не помогает ответу. - - Сосредоточься на указанном файле и отвечай коротко. - Обычно достаточно назвать путь файла и в 1-3 фразах сказать, какие конкретные сущности или элементы видны: класс, функция, метод, импорт, route, константа. - Не используй общие описания файла без конкретных имён. - Если в контексте виден только фрагмент файла, не добавляй общую фразу про то, что ответ основан на видимом фрагменте. Вместо этого просто ограничься тем, что реально видно. - Не превращай ответ в архитектурный обзор проекта. - Не используй секции и подзаголовки. - Если файла нет, ответь одной короткой фразой: `Файл не найден.` - Не придумывай анализ отсутствующего файла. - code_qa_repair_answer: | - Ты исправляешь черновой ответ по результатам проверки groundedness. Вход: draft_answer, validation_reasons, repair_focus, prompt_payload. Исправь только то, на что указывает repair_focus; остальное сохрани. Ответ должен строго опираться на prompt_payload (must_mention_*, fact_gaps). - - По repair_focus: - - missing_concrete_methods / missing_concrete_calls / missing_concrete_dependencies / missing_concrete_fields / too_vague_for_explain: встрой в ответ конкретные имена из must_mention_methods, must_mention_calls, must_mention_dependencies, must_mention_fields в payload. Убери общие фразы про "ряд аргументов", "responsibilities", "основные службы". - - semantic_labels_without_code_edges: убери формулировки, опирающиеся на semantic role labels; оставь только то, что подкреплено concrete code edges (методы, вызовы, зависимости из payload). - - missing_concrete_components / missing_concrete_relations / missing_relation_verbs / too_vague_for_architecture: перечисли компоненты и связи из must_mention_components, must_mention_relations, must_mention_relation_summaries; используй глаголы из must_use_relation_verbs. Не перечисляй только методы. - - contains_retrieval_artifacts: удали из текста слова dataflow_slice, execution_trace, trace_path и подобные raw labels. - - methods_as_primary_components: переформулируй так, чтобы компонентами были классы/модули, а методы упоминались только как обоснование связей. - - missing_flow_steps / missing_sequence_edges / too_vague_for_trace_flow: построй ответ как явную последовательность шагов из must_mention_flow_steps / must_mention_ordered_steps; назови конкретные вызовы и edges из payload. - - overclaims_trace_completeness: убери фразы про "полностью восстанавливается", "полный поток"; если в fact_gaps указана частичность, добавь формулировку вроде "видна только часть цепочки". - Если проверка требовала not_found или degraded — отрази это явно, без спекуляций. - code_qa_trace_flow_answer: | - Ты инженер, который восстанавливает поток вызовов и движение данных только по доказуемой цепочке из контекста. - - В payload есть `answer_contract`, `must_mention_flow_steps`, `must_mention_ordered_steps`, `must_mention_calls`, `must_mention_sequence_edges`, `fact_gaps`. Ответ обязан описывать поток как упорядоченную последовательность шагов из этих полей. Не заявляй полноту потока, если в fact_gaps указано иное. Не делай неподтверждённых утверждений. - - Отвечай только по коду из контекста. Пиши естественным языком, без лишних markdown-секций. - Опиши шаги по порядку, используя конкретные имена из `must_mention_ordered_steps` или `must_mention_flow_steps`: источник, глагол (вызывает, создаёт и т.д.), цель. Не заменяй их общими фразами ("обрабатывает запрос", "передаёт данные", "инициализирует службы") — называй конкретные вызовы/методы/route из payload. - Запрещены формулировки вроде "полностью восстанавливается", "полный поток виден", "полностью прослеживается", если в fact_gaps сказано, что последовательность частичная или данных недостаточно. Если поток частичный — явно скажи об этом в конце. - Не склеивай шаги без прямой связи в payload. Не добавляй шаги, которых нет в must_mention_*. - docs_explain_answer: | - Ты объясняешь документацию системы. - - На вход приходит JSON с полями: - - question - - intent - - sub_intent - - documents - - facts - - relations - - Правила: - - Используй только предоставленные факты - - Не додумывай - - Если данных недостаточно, скажи это явно - - Объясняй структурировано - - Формат ответа: - 1. Краткое описание - 2. Основные элементы - 3. Как это работает - 4. Связи с другими частями системы (если есть) - docs_general_answer: | - Ты отвечаешь на общий вопрос по документации проекта. - - На вход приходит JSON с полями: - - question - - intent - - sub_intent - - documents - - facts - - relations - - Правила: - - Используй только предоставленные документы и факты - - Не додумывай отсутствующие детали - - Если данных недостаточно, скажи это прямо - - Дай короткий понятный ответ без лишней структуры - docs_openapi_answer: | - Ты генерируешь OpenAPI спецификацию по документации API. - - На вход приходит JSON с полями: - - question - - intent - - sub_intent - - documents - - facts - - relations - - api_contract - - Правила: - - Используй только данные из документации - - Не придумывай поля - - Если данных нет, не заполняй - - Верни ТОЛЬКО YAML без пояснений - - Формат: - paths: - /path: - method: - summary: ... - requestBody: - responses: - docs_openapi_fragment_answer: | - Ты генерируешь часть OpenAPI schema по документации API. - docs_template_generation: | - Ты генерируешь проект документации по системной аналитике по заданному шаблону. - - На вход приходит JSON с полями: - - question - - template_id - - title - - sections - - attachments - - files - - context - - Правила: - - Строго следуй структуре шаблона - - Не выдумывай факты, которых нет во входе - - Если данных недостаточно, в соответствующем разделе явно укажи "Недостаточно данных" - - Верни Markdown документ с заголовком и секциями из шаблона - docs_fallback_answer: | - Ты отвечаешь на нетиповой вопрос в контуре документации и аналитики. - - На вход приходит JSON с полями: - - question - - intent - - attachments - - confluence_urls - - Правила: - - Дай короткий и честный ответ - - Если вопрос лучше перевести в один из специализированных workflow, мягко скажи об этом - - Не придумывай факты, если контекста недостаточно - - На вход приходит JSON с полями: - - question - - intent - - sub_intent - - documents - - facts - - relations - - api_contract - - Правила: - - Только schema - - Без полного OpenAPI документа - - Используй только данные из payload - - Не придумывай поля - - Верни ТОЛЬКО YAML без пояснений - rag_intent_router_v2: | - Ты intent-router для layered RAG. - На вход ты получаешь JSON с полями: - - message: текущий запрос пользователя - - active_intent: текущий активный intent диалога или null - - last_query: предыдущий запрос пользователя - - allowed_intents: допустимые intent'ы - - Выбери ровно один intent из allowed_intents и один подходящий sub_intent. - Верни только JSON без markdown и пояснений. - - Строгий формат ответа: - {"intent":"","sub_intent":"","confidence":} - - Правила: - - CODE_QA: объяснение по коду, архитектуре, классам, методам, файлам, блокам кода, поведению приложения по реализации. - - DOCUMENTATION_EXPLAIN: объяснение сущности, компонента, API метода, flow или связанных документов по документации. - - OPENAPI_GENERATION: генерация OpenAPI/Swagger/YAML/spec/schema по документации API. - - GENERAL_QA: слишком общий или нечеткий вопрос по документации/проекту, который нельзя уверенно отнести к explain/openapi. - - GENERATE_DOCS_FROM_CODE: просьба сгенерировать, подготовить или обновить документацию по коду. - - Допустимые docs sub-intents: - - SYSTEM_FLOW_EXPLAIN - - COMPONENT_EXPLAIN - - API_METHOD_EXPLAIN - - ENTITY_EXPLAIN - - RELATED_DOCS_EXPLAIN - - OPENAPI_METHOD_GENERATE - - OPENAPI_FRAGMENT_GENERATE - - GENERIC_QA - - Приоритет: - - Если пользователь просит именно подготовить документацию по коду, выбирай GENERATE_DOCS_FROM_CODE. - - Если есть openapi, swagger, yaml, schema или spec, выбирай OPENAPI_GENERATION. - Если запрос про request, response или schema, выбирай OPENAPI_FRAGMENT_GENERATE. - Иначе выбирай OPENAPI_METHOD_GENERATE. - - Если запрос про связанные документы, где еще описано, что еще посмотреть, какие страницы связаны, какие документы по теме, выбирай DOCUMENTATION_EXPLAIN и sub_intent RELATED_DOCS_EXPLAIN. - - Если есть объясни, как работает, что делает или что такое по документации, выбирай DOCUMENTATION_EXPLAIN. - Для API/endpoint/method выбирай API_METHOD_EXPLAIN. - Для flow/workflow/process выбирай SYSTEM_FLOW_EXPLAIN. - Для entity/сущности выбирай ENTITY_EXPLAIN. - Иначе выбирай COMPONENT_EXPLAIN. - - Если пользователь спрашивает про конкретный класс, файл, метод или блок кода, выбирай CODE_QA. - - Если пользователь спрашивает про README, docs, markdown или конкретную документацию без явного openapi, выбирай DOCUMENTATION_EXPLAIN. - - Если сигнал неочевиден, выбирай GENERAL_QA и confidence <= 0.6. - rag_docs_router_disambiguation_v1: | - Ты помогаешь разрешить неоднозначность в docs sub-intent router. - - На вход приходит JSON: - - query - - normalized_query - - deterministic_primary_candidate - - deterministic_candidates - - ambiguity_reason - - matched_anchor_type - - query_entity_candidates - - query_anchor_candidates - - Выбери ровно один sub_intent только из списка deterministic_candidates. - Не придумывай новые labels. - - Допустимые labels: - - SYSTEM_FLOW_EXPLAIN - - COMPONENT_EXPLAIN - - API_METHOD_EXPLAIN - - ENTITY_EXPLAIN - - RELATED_DOCS_EXPLAIN - - GENERIC_QA - - OPENAPI_METHOD_GENERATE - - OPENAPI_FRAGMENT_GENERATE - - Правила: - - Если запрос обзорный, про структуру документации, "что есть", "с чего начать", выбирай GENERIC_QA. - - Если запрос про runtime health, статус воркера, состояние runtime как понятие/сущность, выбирай ENTITY_EXPLAIN. - - Если запрос про процесс, health check flow, последовательность шагов, выбирай SYSTEM_FLOW_EXPLAIN. - - Если есть точный endpoint/path anchor, не уводи запрос в GENERIC_QA. - - Если есть CamelCase component-like token, предпочитай COMPONENT_EXPLAIN. - - Верни только JSON: - {"sub_intent":"","reason":"","confidence":"low|medium|high"} diff --git a/src/app/modules/agent/orchestration/adapters/intent_router_adapter.py b/src/app/modules/agent/orchestration/adapters/intent_router_adapter.py deleted file mode 100644 index f75cf7a..0000000 --- a/src/app/modules/agent/orchestration/adapters/intent_router_adapter.py +++ /dev/null @@ -1,24 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.observability.module_trace import ModuleTrace -from app.modules.agent.intent_router_v2 import IntentRouterV2 - - -class IntentRouterAdapter: - def __init__(self, router: IntentRouterV2) -> None: - self._router = router - - def route(self, user_query: str, conversation_state, repo_context, trace: ModuleTrace | None = None): - if trace is not None: - trace.log("started", {"question": user_query}) - result = self._router.route(user_query, conversation_state, repo_context) - if trace is not None: - trace.log( - "completed", - { - "intent": result.intent, - "sub_intent": result.query_plan.sub_intent, - "matched_intent_source": result.matched_intent_source, - }, - ) - return result diff --git a/src/app/modules/agent/orchestration/adapters/llm_chat_adapter.py b/src/app/modules/agent/orchestration/adapters/llm_chat_adapter.py deleted file mode 100644 index 32de6ab..0000000 --- a/src/app/modules/agent/orchestration/adapters/llm_chat_adapter.py +++ /dev/null @@ -1,32 +0,0 @@ -from __future__ import annotations - -import asyncio - -from app.modules.agent.observability.module_trace import ModuleTrace -from app.modules.agent.llm.service import AgentLlmService - - -class LlmChatAdapter: - def __init__(self, llm: AgentLlmService, prompt_name: str = "simple_llm_answer") -> None: - self._llm = llm - self._prompt_name = prompt_name - - @property - def prompt_name(self) -> str: - return self._prompt_name - - def build_request(self, message: str, request_id: str) -> dict[str, str]: - return self._llm.build_request( - self._prompt_name, - message, - log_context=f"agent:{request_id}", - ) - - async def generate(self, message: str, request_id: str, trace: ModuleTrace | None = None) -> str: - return await asyncio.to_thread( - self._llm.generate, - self._prompt_name, - message, - log_context=f"agent:{request_id}", - trace=trace, - ) diff --git a/src/app/modules/agent/orchestration/adapters/task_runtime_adapter.py b/src/app/modules/agent/orchestration/adapters/task_runtime_adapter.py deleted file mode 100644 index 42ac762..0000000 --- a/src/app/modules/agent/orchestration/adapters/task_runtime_adapter.py +++ /dev/null @@ -1,44 +0,0 @@ -from __future__ import annotations - -from types import SimpleNamespace - -from app.core.exceptions import AppError -from app.modules.agent.task_runtime.facade import AgentTaskRuntimeFacade -from app.modules.api.domain.models.agent_session import AgentSession -from app.modules.agent.orchestration.context.execution_context import ExecutionContext -from app.schemas.common import ModuleName - - -class TaskRuntimeAdapter: - def __init__(self, runtime: AgentTaskRuntimeFacade) -> None: - self._runtime = runtime - - async def run(self, context: ExecutionContext) -> SimpleNamespace: - rag_session_id = context.session.active_rag_session_id - if not rag_session_id: - raise AppError( - "rag_session_not_bound", - "Agent session has no active rag_session_id for process v2.", - ModuleName.RAG, - ) - - def progress_cb(stage: str, message: str, kind: str = "task_progress", meta: dict | None = None): - payload = dict(meta or {}) - payload.setdefault("kind", kind) - return context.publisher.publish_status( - context.request.request_id, - stage, - message, - payload, - ) - - return await self._runtime.run( - task_id=context.request.request_id, - dialog_session_id=context.session.session_id, - rag_session_id=rag_session_id, - mode="auto", - message=context.request.message, - attachments=[], - files=[], - progress_cb=progress_cb, - ) diff --git a/src/app/modules/agent/orchestration/context/execution_context.py b/src/app/modules/agent/orchestration/context/execution_context.py deleted file mode 100644 index 9db381f..0000000 --- a/src/app/modules/agent/orchestration/context/execution_context.py +++ /dev/null @@ -1,22 +0,0 @@ -from __future__ import annotations - -from dataclasses import dataclass -from typing import Any - -from app.modules.api.domain.models.agent_request import AgentRequest -from app.modules.api.domain.models.agent_session import AgentSession -from app.modules.api.infrastructure.logging.request_trace_logger import RequestTraceLogger -from app.modules.agent.observability.module_trace import RequestTraceContext -from app.modules.agent.orchestration.messaging.client_message_publisher import ClientMessagePublisher - - -@dataclass(slots=True) -class ExecutionContext: - request: AgentRequest - session: AgentSession - publisher: ClientMessagePublisher - trace_logger: RequestTraceLogger - trace: RequestTraceContext - task_context: Any = None - route_result: Any = None - workflow_result: Any = None diff --git a/src/app/modules/agent/orchestration/facade.py b/src/app/modules/agent/orchestration/facade.py deleted file mode 100644 index 25bf6d8..0000000 --- a/src/app/modules/agent/orchestration/facade.py +++ /dev/null @@ -1,72 +0,0 @@ -from __future__ import annotations - -from datetime import datetime, timezone - -from app.core.exceptions import AppError -from app.modules.api.domain.models.agent_request import AgentRequest -from app.modules.api.domain.models.agent_session import AgentSession -from app.modules.api.infrastructure.logging.request_trace_logger import RequestTraceLogger -from app.modules.agent.observability.module_trace import RequestTraceContext -from app.modules.api.infrastructure.stores.in_memory_request_store import InMemoryRequestStore -from app.modules.agent.orchestration.context.execution_context import ExecutionContext -from app.modules.agent.orchestration.messaging.client_message_publisher import ClientMessagePublisher -from app.modules.agent.orchestration.processes.registry import ProcessRegistry -from app.modules.agent.orchestration.runtime.process_runner import ProcessRunner -from app.schemas.common import ErrorPayload, ModuleName -from app.schemas.orchestration import RequestExecutionStatus - - -class OrchestrationFacade: - def __init__( - self, - request_store: InMemoryRequestStore, - process_registry: ProcessRegistry, - process_runner: ProcessRunner, - publisher: ClientMessagePublisher, - trace_logger: RequestTraceLogger, - ) -> None: - self._request_store = request_store - self._process_registry = process_registry - self._process_runner = process_runner - self._publisher = publisher - self._trace_logger = trace_logger - - async def run(self, request: AgentRequest, session: AgentSession) -> None: - try: - process = self._process_registry.get(request.process_version) - if process is None: - raise AppError("process_not_found", f"Unsupported process version: {request.process_version}", ModuleName.AGENT) - request.status = RequestExecutionStatus.RUNNING - self._request_store.save(request) - self._trace_logger.start_request(request, session) - context = ExecutionContext( - request=request, - session=session, - publisher=self._publisher, - trace_logger=self._trace_logger, - trace=RequestTraceContext(request_id=request.request_id, logger=self._trace_logger), - ) - await self._process_runner.run(context, process.steps()) - request.status = RequestExecutionStatus.DONE - request.completed_at = datetime.now(timezone.utc) - self._request_store.save(request) - self._trace_logger.complete_request(request) - except Exception as exc: - request.status = RequestExecutionStatus.ERROR - request.completed_at = datetime.now(timezone.utc) - if isinstance(exc, AppError): - request.error = ErrorPayload(code=exc.code, desc=exc.desc, module=exc.module) - else: - request.error = ErrorPayload( - code="api_runtime_error", - desc="Agent request failed unexpectedly.", - module=ModuleName.AGENT, - ) - self._request_store.save(request) - self._trace_logger.fail_request(request) - await self._publisher.publish_status( - request.request_id, - "orchestrator", - "Во время обработки запроса произошла ошибка.", - {"code": request.error.code}, - ) diff --git a/src/app/modules/agent/orchestration/messaging/client_message_publisher.py b/src/app/modules/agent/orchestration/messaging/client_message_publisher.py deleted file mode 100644 index cdfb20f..0000000 --- a/src/app/modules/agent/orchestration/messaging/client_message_publisher.py +++ /dev/null @@ -1,24 +0,0 @@ -from __future__ import annotations - -from app.modules.api.infrastructure.logging.request_trace_logger import RequestTraceLogger -from app.modules.api.infrastructure.streaming.sse_event_channel import SseEventChannel -from app.modules.agent.orchestration.messaging.status_message_factory import StatusMessageFactory -from app.modules.agent.orchestration.messaging.user_message_factory import UserMessageFactory - - -class ClientMessagePublisher: - def __init__(self, channel: SseEventChannel, trace_logger: RequestTraceLogger) -> None: - self._channel = channel - self._trace_logger = trace_logger - self._status = StatusMessageFactory() - self._user = UserMessageFactory() - - async def publish_status(self, request_id: str, source: str, text: str, payload: dict | None = None) -> None: - event = self._status.create(request_id, source, text, payload) - self._trace_logger.log_event(event) - await self._channel.publish(event) - - async def publish_user(self, request_id: str, source: str, text: str, payload: dict | None = None) -> None: - event = self._user.create(request_id, source, text, payload) - self._trace_logger.log_event(event) - await self._channel.publish(event) diff --git a/src/app/modules/agent/orchestration/messaging/status_message_factory.py b/src/app/modules/agent/orchestration/messaging/status_message_factory.py deleted file mode 100644 index b11befd..0000000 --- a/src/app/modules/agent/orchestration/messaging/status_message_factory.py +++ /dev/null @@ -1,15 +0,0 @@ -from __future__ import annotations - -from app.modules.api.domain.events.client_event import ClientEventRecord -from app.schemas.client_events import ClientEventType - - -class StatusMessageFactory: - def create(self, request_id: str, source: str, text: str, payload: dict | None = None) -> ClientEventRecord: - return ClientEventRecord( - request_id=request_id, - type=ClientEventType.STATUS, - source=source, - text=text, - payload=payload or {}, - ) diff --git a/src/app/modules/agent/orchestration/messaging/user_message_factory.py b/src/app/modules/agent/orchestration/messaging/user_message_factory.py deleted file mode 100644 index aa24fb5..0000000 --- a/src/app/modules/agent/orchestration/messaging/user_message_factory.py +++ /dev/null @@ -1,15 +0,0 @@ -from __future__ import annotations - -from app.modules.api.domain.events.client_event import ClientEventRecord -from app.schemas.client_events import ClientEventType - - -class UserMessageFactory: - def create(self, request_id: str, source: str, text: str, payload: dict | None = None) -> ClientEventRecord: - return ClientEventRecord( - request_id=request_id, - type=ClientEventType.USER, - source=source, - text=text, - payload=payload or {}, - ) diff --git a/src/app/modules/agent/orchestration/processes/registry.py b/src/app/modules/agent/orchestration/processes/registry.py deleted file mode 100644 index 6da53ba..0000000 --- a/src/app/modules/agent/orchestration/processes/registry.py +++ /dev/null @@ -1,14 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.processes.v1.process import V1Process -from app.modules.agent.orchestration.processes.v2.process import V2Process - - -class ProcessRegistry: - def __init__(self, v1_process: V1Process, v2_process: V2Process | None = None) -> None: - self._items = {"v1": v1_process} - if v2_process is not None: - self._items["v2"] = v2_process - - def get(self, version: str): - return self._items.get(version) diff --git a/src/app/modules/agent/orchestration/processes/v1/process.py b/src/app/modules/agent/orchestration/processes/v1/process.py deleted file mode 100644 index b60b575..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/process.py +++ /dev/null @@ -1,9 +0,0 @@ -from __future__ import annotations - - -class V1Process: - def __init__(self, steps: list) -> None: - self._steps = steps - - def steps(self) -> list: - return list(self._steps) diff --git a/src/app/modules/agent/orchestration/processes/v1/prompt_payload_builder.py b/src/app/modules/agent/orchestration/processes/v1/prompt_payload_builder.py deleted file mode 100644 index 9faf40e..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/prompt_payload_builder.py +++ /dev/null @@ -1,11 +0,0 @@ -from __future__ import annotations - -import json - -from app.modules.agent.orchestration.context.execution_context import ExecutionContext - - -class V1PromptPayloadBuilder: - def build(self, context: ExecutionContext) -> str: - payload = {"question": context.request.message} - return json.dumps(payload, ensure_ascii=False, indent=2) diff --git a/src/app/modules/agent/orchestration/processes/v1/prompts.yml b/src/app/modules/agent/orchestration/processes/v1/prompts.yml deleted file mode 100644 index 243f86c..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/prompts.yml +++ /dev/null @@ -1,12 +0,0 @@ -prompts: - simple_llm_answer: | - Ты полезный AI-ассистент проекта. - - На вход приходит JSON с полем: - - question - - Правила: - - Отвечай как персонаж мемов из дагестана - - Если вопрос неясный, аккуратно укажи, чего не хватает - - Не выдумывай несуществующие факты о проекте - - Формулируй ответ как обычное сообщение пользователю diff --git a/src/app/modules/agent/orchestration/processes/v1/simple_llm_workflow.py b/src/app/modules/agent/orchestration/processes/v1/simple_llm_workflow.py deleted file mode 100644 index 62ef2bf..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/simple_llm_workflow.py +++ /dev/null @@ -1,42 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.adapters.llm_chat_adapter import LlmChatAdapter -from app.modules.agent.orchestration.context.execution_context import ExecutionContext -from app.modules.agent.orchestration.processes.v1.prompt_payload_builder import V1PromptPayloadBuilder - - -class SimpleLlmWorkflow: - workflow_id = "simple_llm" - - def __init__( - self, - llm: LlmChatAdapter, - prompt_payload_builder: V1PromptPayloadBuilder | None = None, - ) -> None: - self._llm = llm - self._payload_builder = prompt_payload_builder or V1PromptPayloadBuilder() - - async def run(self, context: ExecutionContext) -> dict: - workflow_trace = context.trace.module("task_workflow") - workflow_trace.log("started", {"workflow_id": self.workflow_id}) - prompt_payload = self._payload_builder.build(context) - answer = await self._llm.generate( - prompt_payload, - context.request.request_id, - trace=context.trace.module("llm"), - ) - result = { - "workflow_id": self.workflow_id, - "prompt_name": self._llm.prompt_name, - "prompt_payload": prompt_payload, - "answer": answer, - } - workflow_trace.log( - "completed", - { - "workflow_id": self.workflow_id, - "prompt_name": self._llm.prompt_name, - "answer_length": len(answer), - }, - ) - return result diff --git a/src/app/modules/agent/orchestration/processes/v1/steps/bootstrap_step.py b/src/app/modules/agent/orchestration/processes/v1/steps/bootstrap_step.py deleted file mode 100644 index 34e344d..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/steps/bootstrap_step.py +++ /dev/null @@ -1,23 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.context.execution_context import ExecutionContext - - -class BootstrapStep: - async def run(self, context: ExecutionContext) -> None: - context.trace.module("orchestrator").log( - "bootstrap", - {"status": "started", "process_version": context.request.process_version}, - ) - await context.publisher.publish_status( - context.request.request_id, - "orchestrator", - "Запрос принят и поставлен в обработку.", - ) - await context.publisher.publish_status( - context.request.request_id, - "orchestrator", - "Запускаю процесс обработки v1.", - {"process_version": context.request.process_version}, - ) - context.trace.module("orchestrator").log("bootstrap", {"status": "completed"}) diff --git a/src/app/modules/agent/orchestration/processes/v1/steps/execute_llm_workflow_step.py b/src/app/modules/agent/orchestration/processes/v1/steps/execute_llm_workflow_step.py deleted file mode 100644 index 61d7d1a..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/steps/execute_llm_workflow_step.py +++ /dev/null @@ -1,35 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.context.execution_context import ExecutionContext -from app.modules.agent.orchestration.processes.v1.simple_llm_workflow import SimpleLlmWorkflow - - -class ExecuteLlmWorkflowStep: - def __init__(self, workflow: SimpleLlmWorkflow) -> None: - self._workflow = workflow - - async def run(self, context: ExecutionContext) -> None: - request = context.request - await context.publisher.publish_status( - request.request_id, - "task_workflow", - f"Запускаю workflow {self._workflow.workflow_id}.", - ) - await context.publisher.publish_status( - request.request_id, - "prompt_builder", - "Формирую prompt payload для LLM.", - ) - result = await self._workflow.run(context) - request.answer = str(result.get("answer") or "") - context.workflow_result = result - await context.publisher.publish_status( - request.request_id, - "llm_process", - "Ответ от LLM получен.", - { - "workflow_id": result.get("workflow_id"), - "prompt_name": result.get("prompt_name"), - "answer_length": len(request.answer), - }, - ) diff --git a/src/app/modules/agent/orchestration/processes/v1/steps/finalize_step.py b/src/app/modules/agent/orchestration/processes/v1/steps/finalize_step.py deleted file mode 100644 index 68afc79..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/steps/finalize_step.py +++ /dev/null @@ -1,20 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.context.execution_context import ExecutionContext - - -class FinalizeStep: - async def run(self, context: ExecutionContext) -> None: - request = context.request - context.trace.module("orchestrator").log("finalize", {"status": "started"}) - await context.publisher.publish_user( - request.request_id, - "agent", - request.answer or "", - ) - await context.publisher.publish_status( - request.request_id, - "orchestrator", - "Обработка запроса завершена.", - ) - context.trace.module("orchestrator").log("finalize", {"status": "completed"}) diff --git a/src/app/modules/agent/orchestration/processes/v1/steps/run_llm_step.py b/src/app/modules/agent/orchestration/processes/v1/steps/run_llm_step.py deleted file mode 100644 index 42c3c74..0000000 --- a/src/app/modules/agent/orchestration/processes/v1/steps/run_llm_step.py +++ /dev/null @@ -1,26 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.adapters.llm_chat_adapter import LlmChatAdapter -from app.modules.agent.orchestration.context.execution_context import ExecutionContext - - -class RunLlmStep: - def __init__(self, llm: LlmChatAdapter) -> None: - self._llm = llm - - async def run(self, context: ExecutionContext) -> None: - request = context.request - context.trace_logger.log_step(request.request_id, "run_llm", "started") - await context.publisher.publish_status( - request.request_id, - "llm_process", - "Отправляю запрос пользователя в LLM.", - ) - answer = await self._llm.generate(request.message, request.request_id) - request.answer = answer - await context.publisher.publish_status( - request.request_id, - "llm_process", - "Ответ от LLM получен.", - ) - context.trace_logger.log_step(request.request_id, "run_llm", "completed", {"answer_length": len(answer)}) diff --git a/src/app/modules/agent/orchestration/processes/v2/README.md b/src/app/modules/agent/orchestration/processes/v2/README.md deleted file mode 100644 index 0a7d516..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/README.md +++ /dev/null @@ -1,189 +0,0 @@ -# Process V2 - -`v2` is the current default orchestration process for agent requests. - -It is designed as a small stage-based pipeline: - -1. accept request -2. route intent -3. run intent-specific workflow -4. publish final user response - -The process definition lives in [process.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v2/process.py). - -## Step Map - -### 1. Bootstrap - -File: -[bootstrap_step.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v1/steps/bootstrap_step.py) - -Responsibility: -- announce request acceptance -- announce selected process version -- initialize trace logging - -SSE source: -- `orchestrator` - -### 2. Intent Router - -File: -[route_intent_step.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v2/steps/route_intent_step.py) - -Responsibility: -- build task runtime context -- enrich request context -- call `IntentRouterV2` -- persist `route_result` into orchestration context - -SSE source: -- `intent_router` - -Main status messages: -- request is being routed -- final `intent / sub_intent` - -### 3. Workflow Execution - -Base behavior: -[workflow_step_base.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v2/steps/workflow_step_base.py) - -Intent-specific steps: -- [execute_documentation_workflow_step.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v2/steps/execute_documentation_workflow_step.py) -- [execute_openapi_workflow_step.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v2/steps/execute_openapi_workflow_step.py) -- [execute_general_qa_workflow_step.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v2/steps/execute_general_qa_workflow_step.py) -- [execute_fallback_workflow_step.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v2/steps/execute_fallback_workflow_step.py) - -Responsibility: -- choose workflow by `route_result.intent` -- run the selected workflow -- expose workflow diagnostics through SSE -- store workflow result in orchestration context - -Workflow mapping: - -| Intent | Workflow class | Notes | -|---|---|---| -| `DOCUMENTATION_EXPLAIN` | `DocsQaWorkflow` | docs explanation pipeline | -| `OPENAPI_GENERATION` | `OpenApiWorkflow` | OpenAPI generation from docs evidence | -| `GENERAL_QA` | `GeneralQaWorkflow` | general docs-oriented QA | -| any other intent | `FallbackWorkflow` | safety fallback | - -Published SSE stages: -- `task_workflow` -- `rag_retrieval` -- `evidence_gate` -- `workflow_result` - -### 4. Finalize - -File: -[finalize_step.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/processes/v1/steps/finalize_step.py) - -Responsibility: -- publish final `user` event with answer text -- publish terminal status message -- let facade mark request as `done` - -SSE sources: -- `agent` -- `orchestrator` - -## Intent Routing - -Router adapter: -[intent_router_adapter.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/orchestration/adapters/intent_router_adapter.py) - -Underlying implementation: -[router.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/agent/intent_router_v2/router.py) - -The router decides: -- `intent` -- `sub_intent` -- retrieval profile -- retrieval layers and constraints -- evidence policy - -For docs-oriented requests the main active intents are: -- `DOCUMENTATION_EXPLAIN` -- `OPENAPI_GENERATION` -- `GENERAL_QA` - -## Retrieval And Evidence Gate - -`v2` does not implement retrieval itself inside orchestration. - -Instead it delegates execution to workflow classes that internally use the existing docs pipeline: - -- [docs_qa.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/agent/task_runtime/workflows/docs_qa.py) -- [openapi.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/agent/task_runtime/workflows/openapi.py) -- [general_qa.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/agent/task_runtime/workflows/general_qa.py) - -These workflows currently rely on: -- `DocsQAPipelineRunner` -- docs retrieval planning -- docs evidence builder -- docs gate logic - -Important: -- docs retrieval and gate logic still live in the existing docs runtime layer -- orchestration `v2` only makes those stages explicit at transport/process level - -## SSE Contract In V2 - -Client-visible message categories: -- `status` -- `user` -- `system` - -Currently used in `v2`: - -- `status` from `orchestrator` -- `status` from `intent_router` -- `status` from `task_workflow` -- `status` from `rag_retrieval` -- `status` from `evidence_gate` -- `status` from `workflow_result` -- `user` from `agent` - -Typical sequence: - -1. request accepted -2. process `v2` started -3. routing started -4. route selected -5. workflow started -6. retrieval diagnostics -7. evidence gate diagnostics -8. workflow result summary -9. final user answer -10. processing completed - -## Trace Logging - -Per-request trace files are written by: -[request_trace_logger.py](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/src/app/modules/api/infrastructure/logging/request_trace_logger.py) - -Location: -[runtime_traces/agent_requests](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/runtime_traces/agent_requests) - -Each request trace contains: -- request metadata -- user message -- process steps -- emitted events -- final result or error - -## Extension Points - -Recommended next extensions: - -1. add a dedicated workflow step for `GENERATE_DOCS_FROM_CODE` -2. move docs gate diagnostics into strongly typed orchestration payloads -3. split workflow execution into smaller orchestration stages: - - retrieval - - evidence assembly - - gate - - answer generation -4. add session-level trace aggregation in addition to request-level logs diff --git a/src/app/modules/agent/orchestration/processes/v2/process.py b/src/app/modules/agent/orchestration/processes/v2/process.py deleted file mode 100644 index 908ed23..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/process.py +++ /dev/null @@ -1,9 +0,0 @@ -from __future__ import annotations - - -class V2Process: - def __init__(self, steps: list) -> None: - self._steps = steps - - def steps(self) -> list: - return list(self._steps) diff --git a/src/app/modules/agent/orchestration/processes/v2/prompt_payload_builder.py b/src/app/modules/agent/orchestration/processes/v2/prompt_payload_builder.py deleted file mode 100644 index 223e3b8..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/prompt_payload_builder.py +++ /dev/null @@ -1,35 +0,0 @@ -from __future__ import annotations - -import json - -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle, OpenAPIResult - - -class V2PromptPayloadBuilder: - def build( - self, - *, - question: str, - intent: str, - sub_intent: str, - evidence_bundle: DocsEvidenceBundle, - api_contract: OpenAPIResult | None = None, - ) -> str: - payload = { - "question": question, - "documents": list(evidence_bundle.documents), - "facts": list(evidence_bundle.facts), - "entities": list(evidence_bundle.entities), - "workflows": list(evidence_bundle.workflows), - "relations": list(evidence_bundle.relations), - "chunks": list(evidence_bundle.chunks), - } - if api_contract is not None: - payload["api_contract"] = { - "path": api_contract.path, - "method": api_contract.method, - "request_schema": api_contract.request_schema, - "response_schema": api_contract.response_schema, - "diagnostics": dict(api_contract.diagnostics), - } - return json.dumps(payload, ensure_ascii=False, indent=2) diff --git a/src/app/modules/agent/orchestration/processes/v2/prompt_selector.py b/src/app/modules/agent/orchestration/processes/v2/prompt_selector.py deleted file mode 100644 index 16714f6..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/prompt_selector.py +++ /dev/null @@ -1,18 +0,0 @@ -"""Выбор prompt для process-local workflow v2.""" - -from __future__ import annotations - - -class V2PromptSelector: - _DOCS_INTENT_PROMPTS = { - "DOCUMENTATION_EXPLAIN": "documentation_explain_answer", - "GENERAL_QA": "general_qa_answer", - } - - def select(self, *, intent: str = "DOCUMENTATION_EXPLAIN", sub_intent: str, answer_mode: str) -> str: - intent_key = (intent or "DOCUMENTATION_EXPLAIN").upper() - if intent_key in {"OPENAPI_GENERATION", "OPENAPI_FROM_DOCUMENTATION"}: - if sub_intent.upper() == "OPENAPI_FRAGMENT_GENERATE": - return "openapi_fragment_answer" - return "openapi_answer" - return self._DOCS_INTENT_PROMPTS.get(intent_key, "documentation_explain_answer") diff --git a/src/app/modules/agent/orchestration/processes/v2/prompts.yml b/src/app/modules/agent/orchestration/processes/v2/prompts.yml deleted file mode 100644 index 5142637..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/prompts.yml +++ /dev/null @@ -1,96 +0,0 @@ -prompts: - documentation_explain_answer: | - Ты объясняешь документацию системы. - - На вход приходит JSON с полями: - - question - - documents - - facts - - entities - - workflows - - relations - - chunks - - Правила: - - Используй только предоставленные факты - - Не додумывай - - Если данных недостаточно, скажи это явно - - Объясняй структурировано - - Формат ответа: - 1. Краткое описание - 2. Основные элементы - 3. Как это работает - 4. Связи с другими частями системы (если есть) - general_qa_answer: | - Ты отвечаешь на общий вопрос по документации проекта. - - На вход приходит JSON с полями: - - question - - documents - - facts - - entities - - workflows - - relations - - chunks - - Правила: - - Используй только предоставленные документы и факты - - Не додумывай отсутствующие детали - - Если данных недостаточно, скажи это прямо - - Дай короткий понятный ответ без лишней структуры - openapi_answer: | - Ты генерируешь OpenAPI спецификацию по документации API. - - На вход приходит JSON с полями: - - question - - documents - - facts - - entities - - workflows - - relations - - chunks - - api_contract - - Правила: - - Используй только данные из документации - - Не придумывай поля - - Если данных нет, не заполняй - - Верни ТОЛЬКО YAML без пояснений - - Формат: - paths: - /path: - method: - summary: ... - requestBody: - responses: - openapi_fragment_answer: | - Ты генерируешь часть OpenAPI schema по документации API. - - На вход приходит JSON с полями: - - question - - documents - - facts - - entities - - workflows - - relations - - chunks - - api_contract - - Правила: - - Используй только данные из документации - - Не придумывай отсутствующие поля - - Верни только содержимое нужного фрагмента - fallback_answer: | - Ты формируешь безопасный fallback-ответ. - - На вход приходит JSON с полями: - - question - - attachments - - confluence_urls - - Правила: - - Если специализированный workflow не выбран, честно скажи об ограничении - - Используй только данные из payload - - Не выдумывай детали проекта diff --git a/src/app/modules/agent/orchestration/processes/v2/steps/execute_documentation_workflow_step.py b/src/app/modules/agent/orchestration/processes/v2/steps/execute_documentation_workflow_step.py deleted file mode 100644 index 1268038..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/steps/execute_documentation_workflow_step.py +++ /dev/null @@ -1,7 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.processes.v2.steps.workflow_step_base import WorkflowStepBase - - -class ExecuteDocumentationWorkflowStep(WorkflowStepBase): - intent_name = "DOCUMENTATION_EXPLAIN" diff --git a/src/app/modules/agent/orchestration/processes/v2/steps/execute_fallback_workflow_step.py b/src/app/modules/agent/orchestration/processes/v2/steps/execute_fallback_workflow_step.py deleted file mode 100644 index 6bf3c1f..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/steps/execute_fallback_workflow_step.py +++ /dev/null @@ -1,18 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.context.execution_context import ExecutionContext -from app.modules.agent.orchestration.processes.v2.steps.workflow_step_base import WorkflowStepBase - - -class ExecuteFallbackWorkflowStep(WorkflowStepBase): - intent_name = "FALLBACK" - - def should_run(self, context: ExecutionContext) -> bool: - route_result = context.route_result - if route_result is None: - return False - return str(route_result.intent or "").upper() not in { - "DOCUMENTATION_EXPLAIN", - "OPENAPI_GENERATION", - "GENERAL_QA", - } diff --git a/src/app/modules/agent/orchestration/processes/v2/steps/execute_general_qa_workflow_step.py b/src/app/modules/agent/orchestration/processes/v2/steps/execute_general_qa_workflow_step.py deleted file mode 100644 index f7893fd..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/steps/execute_general_qa_workflow_step.py +++ /dev/null @@ -1,7 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.processes.v2.steps.workflow_step_base import WorkflowStepBase - - -class ExecuteGeneralQaWorkflowStep(WorkflowStepBase): - intent_name = "GENERAL_QA" diff --git a/src/app/modules/agent/orchestration/processes/v2/steps/execute_openapi_workflow_step.py b/src/app/modules/agent/orchestration/processes/v2/steps/execute_openapi_workflow_step.py deleted file mode 100644 index 755f6c1..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/steps/execute_openapi_workflow_step.py +++ /dev/null @@ -1,7 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.processes.v2.steps.workflow_step_base import WorkflowStepBase - - -class ExecuteOpenApiWorkflowStep(WorkflowStepBase): - intent_name = "OPENAPI_GENERATION" diff --git a/src/app/modules/agent/orchestration/processes/v2/steps/route_intent_step.py b/src/app/modules/agent/orchestration/processes/v2/steps/route_intent_step.py deleted file mode 100644 index e02990b..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/steps/route_intent_step.py +++ /dev/null @@ -1,71 +0,0 @@ -from __future__ import annotations - -import asyncio - -from app.core.exceptions import AppError -from app.modules.agent.task_runtime.context import TaskRuntimeContextBuilder -from app.modules.agent.task_runtime.enrichment import ContextEnrichmentService -from app.modules.agent.orchestration.adapters.intent_router_adapter import IntentRouterAdapter -from app.modules.agent.orchestration.context.execution_context import ExecutionContext -from app.modules.agent.orchestration.v2_progress import build_progress_callback -from app.schemas.common import ModuleName - - -class RouteIntentStep: - def __init__( - self, - router: IntentRouterAdapter, - context_builder: TaskRuntimeContextBuilder, - enrichment: ContextEnrichmentService, - ) -> None: - self._router = router - self._context_builder = context_builder - self._enrichment = enrichment - - async def run(self, context: ExecutionContext) -> None: - rag_session_id = context.session.active_rag_session_id - if not rag_session_id: - raise AppError( - "rag_session_not_bound", - "Agent session has no active rag_session_id for process v2.", - ModuleName.RAG, - ) - request = context.request - loop = asyncio.get_running_loop() - task_context = self._context_builder.build( - task_id=request.request_id, - dialog_session_id=context.session.session_id, - rag_session_id=rag_session_id, - mode="auto", - message=request.message, - attachments=[], - files=[], - progress_cb=build_progress_callback(loop, context.publisher, request.request_id), - trace=context.trace, - ) - task_context.enriched_context = self._enrichment.enrich(task_context) - context.task_context = task_context - await context.publisher.publish_status( - request.request_id, - "intent_router", - "Маршрутизирую запрос и определяю целевой workflow.", - ) - route_result = await asyncio.to_thread( - self._router.route, - request.message, - task_context.conversation_state, - task_context.repo_context, - context.trace.module("intent_router"), - ) - task_context.route_result = route_result - context.route_result = route_result - await context.publisher.publish_status( - request.request_id, - "intent_router", - f"Маршрут выбран: {route_result.intent} / {route_result.query_plan.sub_intent}.", - { - "intent": route_result.intent, - "sub_intent": route_result.query_plan.sub_intent, - "matched_intent_source": route_result.matched_intent_source, - }, - ) diff --git a/src/app/modules/agent/orchestration/processes/v2/steps/run_task_workflow_step.py b/src/app/modules/agent/orchestration/processes/v2/steps/run_task_workflow_step.py deleted file mode 100644 index f8d0a1b..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/steps/run_task_workflow_step.py +++ /dev/null @@ -1,30 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.adapters.task_runtime_adapter import TaskRuntimeAdapter -from app.modules.agent.orchestration.context.execution_context import ExecutionContext - - -class RunTaskWorkflowStep: - def __init__(self, runtime: TaskRuntimeAdapter) -> None: - self._runtime = runtime - - async def run(self, context: ExecutionContext) -> None: - request = context.request - context.trace_logger.log_step(request.request_id, "task_runtime", "started") - await context.publisher.publish_status( - request.request_id, - "orchestrator", - "Запускаю pipeline v2: intent router, rag retrieval и task workflow.", - {"process_version": request.process_version}, - ) - result = await self._runtime.run(context) - request.answer = result.answer or "" - context.trace_logger.log_step( - request.request_id, - "task_runtime", - "completed", - { - "result_type": getattr(result.result_type, "value", str(result.result_type)), - "meta": getattr(result, "meta", {}) or {}, - }, - ) diff --git a/src/app/modules/agent/orchestration/processes/v2/steps/workflow_step_base.py b/src/app/modules/agent/orchestration/processes/v2/steps/workflow_step_base.py deleted file mode 100644 index bbd26ce..0000000 --- a/src/app/modules/agent/orchestration/processes/v2/steps/workflow_step_base.py +++ /dev/null @@ -1,68 +0,0 @@ -from __future__ import annotations - -import asyncio -from typing import Any - -from app.modules.agent.orchestration.context.execution_context import ExecutionContext - - -class WorkflowStepBase: - intent_name = "" - - def __init__(self, workflow, step_name: str) -> None: - self._workflow = workflow - self._step_name = step_name - - def should_run(self, context: ExecutionContext) -> bool: - route_result = context.route_result - return bool(route_result) and str(route_result.intent or "").upper() == self.intent_name - - async def run(self, context: ExecutionContext) -> None: - request = context.request - task_context = context.task_context - await context.publisher.publish_status( - request.request_id, - "task_workflow", - f"Запускаю workflow {self._workflow.workflow_id}.", - {"intent": context.route_result.intent, "sub_intent": context.route_result.query_plan.sub_intent}, - ) - result = await asyncio.to_thread(self._workflow.run, task_context) - context.workflow_result = result - request.answer = result.answer or "" - diagnostics = dict(result.meta.get("diagnostics") or {}) - await self._publish_diagnostics(context, diagnostics, result) - - async def _publish_diagnostics(self, context: ExecutionContext, diagnostics: dict[str, Any], result: Any) -> None: - request_id = context.request.request_id - if diagnostics: - await context.publisher.publish_status( - request_id, - "rag_retrieval", - "RAG retrieval завершен.", - { - "planned_layers": list(diagnostics.get("planned_layers") or diagnostics.get("layers_used") or []), - "executed_layers": list(diagnostics.get("executed_layers") or []), - "non_empty_layers": list(diagnostics.get("non_empty_layers") or diagnostics.get("docs_layers_with_hits") or []), - }, - ) - await context.publisher.publish_status( - request_id, - "evidence_gate", - "Evidence gate оценен.", - { - "decision": diagnostics.get("gate_decision"), - "reason": diagnostics.get("gate_decision_reason"), - "missing": list(diagnostics.get("gate_missing_requirements") or []), - "satisfied": list(diagnostics.get("gate_satisfied_requirements") or []), - }, - ) - await context.publisher.publish_status( - request_id, - "workflow_result", - f"Workflow {self._workflow.workflow_id} завершен.", - { - "workflow_id": self._workflow.workflow_id, - "result_type": getattr(result.result_type, "value", str(result.result_type)), - "answer_length": len(result.answer or ""), - }, - ) diff --git a/src/app/modules/agent/orchestration/runtime/process_runner.py b/src/app/modules/agent/orchestration/runtime/process_runner.py deleted file mode 100644 index 2bed04a..0000000 --- a/src/app/modules/agent/orchestration/runtime/process_runner.py +++ /dev/null @@ -1,12 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.orchestration.context.execution_context import ExecutionContext - - -class ProcessRunner: - async def run(self, context: ExecutionContext, steps: list) -> None: - for step in steps: - should_run = getattr(step, "should_run", None) - if callable(should_run) and not should_run(context): - continue - await step.run(context) diff --git a/src/app/modules/agent/orchestration/v2_progress.py b/src/app/modules/agent/orchestration/v2_progress.py deleted file mode 100644 index 7d1ade4..0000000 --- a/src/app/modules/agent/orchestration/v2_progress.py +++ /dev/null @@ -1,17 +0,0 @@ -from __future__ import annotations - -import asyncio - -from app.modules.agent.orchestration.messaging.client_message_publisher import ClientMessagePublisher - - -def build_progress_callback(loop: asyncio.AbstractEventLoop, publisher: ClientMessagePublisher, request_id: str): - def progress_cb(stage: str, message: str, kind: str = "task_progress", meta: dict | None = None) -> None: - payload = dict(meta or {}) - payload.setdefault("kind", kind) - asyncio.run_coroutine_threadsafe( - publisher.publish_status(request_id, stage, message, payload), - loop, - ) - - return progress_cb diff --git a/src/app/modules/agent/runtime/__init__.py b/src/app/modules/agent/runtime/__init__.py deleted file mode 100644 index a0b10ff..0000000 --- a/src/app/modules/agent/runtime/__init__.py +++ /dev/null @@ -1,22 +0,0 @@ -"""Публичный API runtime: оркестрация роутинг → retrieval → evidence gate → генерация ответа.""" - -from app.modules.agent.runtime.executor import AgentRuntimeExecutor -from app.modules.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner -from app.modules.agent.runtime.models import ( - RuntimeDraftAnswer, - RuntimeExecutionState, - RuntimeFinalResult, -) -from app.modules.agent.runtime.steps.gates.post.post_gate import RuntimeValidationResult -from app.modules.agent.runtime.steps.retrieval import RuntimeRepoContextFactory, RuntimeRetrievalAdapter - -__all__ = [ - "AgentRuntimeExecutor", - "DocsQAPipelineRunner", - "RuntimeDraftAnswer", - "RuntimeExecutionState", - "RuntimeFinalResult", - "RuntimeRepoContextFactory", - "RuntimeRetrievalAdapter", - "RuntimeValidationResult", -] diff --git a/src/app/modules/agent/runtime/code_qa_runner_adapter.py b/src/app/modules/agent/runtime/code_qa_runner_adapter.py deleted file mode 100644 index 8c2f5e3..0000000 --- a/src/app/modules/agent/runtime/code_qa_runner_adapter.py +++ /dev/null @@ -1,70 +0,0 @@ -"""Адаптер AgentRuntimeExecutor к протоколу AgentRunner для интеграции с chat-слоем.""" - -from __future__ import annotations - -import asyncio -import logging - -from app.modules.agent.runtime import AgentRuntimeExecutor -from app.modules.contracts import AgentRunner -from app.schemas.chat import TaskResultType - -LOGGER = logging.getLogger(__name__) - - -class CodeQaRunnerAdapter: - """Реализация AgentRunner через AgentRuntimeExecutor (sync execute в executor).""" - - def __init__(self, executor: AgentRuntimeExecutor) -> None: - self._executor = executor - - async def run( - self, - *, - task_id: str, - dialog_session_id: str, - rag_session_id: str, - mode: str, - message: str, - attachments: list[dict], - files: list[dict], - progress_cb=None, - ): - files_map = _files_to_map(files) - loop = asyncio.get_running_loop() - result = await loop.run_in_executor( - None, - lambda: self._executor.execute( - user_query=message, - rag_session_id=rag_session_id, - files_map=files_map, - ), - ) - return _to_agent_run_result(result, task_id=task_id) - - -def _files_to_map(files: list[dict]) -> dict[str, dict]: - out: dict[str, dict] = {} - for item in files or []: - if isinstance(item, dict) and item.get("path"): - out[str(item["path"])] = dict(item) - return out - - -def _to_agent_run_result(final, *, task_id: str): - from types import SimpleNamespace - - meta = {} - if final: - meta = { - "task_id": task_id, - "answer_mode": getattr(final, "answer_mode", "normal"), - } - if getattr(final, "diagnostics", None) is not None: - meta["diagnostics"] = getattr(final.diagnostics, "model_dump", lambda **kw: {})(mode="json") - return SimpleNamespace( - result_type=TaskResultType.ANSWER, - answer=final.final_answer if final else "", - changeset=[], - meta=meta, - ) diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/__init__.py b/src/app/modules/agent/runtime/docs_qa_pipeline/__init__.py deleted file mode 100644 index 0da2da5..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/__init__.py +++ /dev/null @@ -1,9 +0,0 @@ -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsDiagnostics, DocsQAPipelineResult, OpenAPIResult -from app.modules.agent.runtime.docs_qa_pipeline.pipeline import DocsQAPipelineRunner - -__all__ = [ - "DocsDiagnostics", - "DocsQAPipelineRunner", - "DocsQAPipelineResult", - "OpenAPIResult", -] diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/anchor_selector.py b/src/app/modules/agent/runtime/docs_qa_pipeline/anchor_selector.py deleted file mode 100644 index fc87747..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/anchor_selector.py +++ /dev/null @@ -1,56 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.docs_qa_pipeline.doc_identity import DocsCanonicalDocIdResolver - - -class DocsAnchorSelector: - def __init__(self, resolver: DocsCanonicalDocIdResolver | None = None) -> None: - self._resolver = resolver or DocsCanonicalDocIdResolver() - - def select(self, *, sub_intent: str, anchor_type: str, anchor_value: str | None, rows: list[dict]) -> dict[str, object]: - if sub_intent != "RELATED_DOCS_EXPLAIN": - return {"anchor_candidates": [], "selected_anchor": None, "anchor_selection_reason": "", "anchor_match_type": ""} - scored = self._scored_candidates(anchor_type=anchor_type, anchor_value=anchor_value, rows=rows) - if not scored: - return {"anchor_candidates": [], "selected_anchor": None, "anchor_selection_reason": "", "anchor_match_type": ""} - selected = scored[0] - return { - "anchor_candidates": [item["doc_id"] for item in scored], - "selected_anchor": selected["doc_id"], - "anchor_selection_reason": selected["reason"], - "anchor_match_type": selected["match_type"], - } - - def _scored_candidates(self, *, anchor_type: str, anchor_value: str | None, rows: list[dict]) -> list[dict[str, object]]: - value = str(anchor_value or "").strip().lower() - endpoint_slug = value.strip("/").replace("/", "_").replace("-", "_") - scored: list[dict[str, object]] = [] - for row in rows: - metadata = dict(row.get("metadata") or {}) - for doc_id in self._resolver.candidates(row): - score = 0 - match_type = "semantic_fallback" - reason = "relation_neighbor" - if anchor_type == "endpoint" and value: - if str(metadata.get("endpoint") or "").strip().lower() == value: - score, match_type, reason = 100, "exact_path", "metadata.endpoint exact match" - elif endpoint_slug and endpoint_slug in doc_id.lower(): - score, match_type, reason = 90, "exact_path", "doc_id matches requested endpoint slug" - elif endpoint_slug and endpoint_slug in str(row.get("path") or "").lower().replace("-", "_"): - score, match_type, reason = 85, "exact_path", "path matches requested endpoint slug" - elif anchor_type == "entity" and value: - entity_value = str(metadata.get("entity") or "").strip().lower() - if entity_value == value: - score, match_type, reason = 100, "exact_entity", "metadata.entity exact match" - elif value in doc_id.lower(): - score, match_type, reason = 90, "exact_title", "doc_id matches requested entity" - if score == 0: - score = 40 if str(row.get("layer") or "") == "D5_RELATION_GRAPH" else 20 - scored.append({"doc_id": doc_id, "score": score, "match_type": match_type, "reason": reason}) - unique: dict[str, dict[str, object]] = {} - for item in scored: - doc_id = str(item["doc_id"]) - current = unique.get(doc_id) - if current is None or int(item["score"]) > int(current["score"]): - unique[doc_id] = item - return sorted(unique.values(), key=lambda item: (-int(item["score"]), str(item["doc_id"]))) diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/answer_synthesizer.py b/src/app/modules/agent/runtime/docs_qa_pipeline/answer_synthesizer.py deleted file mode 100644 index 8e83fec..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/answer_synthesizer.py +++ /dev/null @@ -1,43 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle - - -class DocsAnswerSynthesizer: - def synthesize(self, query: str, evidence_bundle: DocsEvidenceBundle) -> str: - mode = "graph_summary" if evidence_bundle.sub_intent == "RELATED_DOCS_EXPLAIN" else "prose" - if mode == "list": - return self._list_answer(evidence_bundle) - if mode == "graph_summary": - return self._graph_summary(evidence_bundle) - return self._prose_answer(query, evidence_bundle) - - def _prose_answer(self, query: str, bundle: DocsEvidenceBundle) -> str: - parts = [item["content"] for item in bundle.facts[:3] if item.get("content")] - if not parts: - parts = [item["content"] for item in bundle.entities[:2] if item.get("content")] - if not parts: - parts = [item["content"] for item in bundle.workflows[:2] if item.get("content")] - if not parts: - parts = [item["content"] for item in bundle.documents[:2] if item.get("content")] - if not parts: - parts = [item["content"] for item in bundle.chunks[:2] if item.get("content")] - if not parts: - return f"Недостаточно данных в документации для ответа на запрос: {query}" - return "\n".join(parts) - - def _list_answer(self, bundle: DocsEvidenceBundle) -> str: - items = [item["title"] or item["path"] for item in bundle.documents[:5]] - if not items: - items = [item["title"] or item["path"] for item in bundle.facts[:5]] - if not items: - return "Подходящие документы не найдены." - return "\n".join(f"- {item}" for item in items) - - def _graph_summary(self, bundle: DocsEvidenceBundle) -> str: - items = [item["title"] or item["content"] for item in bundle.relations[:5]] - if not items: - items = [item["title"] or item["path"] for item in bundle.documents[:5]] - if not items: - return "Связанные документы не найдены." - return "\n".join(f"- {item}" for item in items) diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/diagnostics_builder.py b/src/app/modules/agent/runtime/docs_qa_pipeline/diagnostics_builder.py deleted file mode 100644 index 1d18937..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/diagnostics_builder.py +++ /dev/null @@ -1,259 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.docs_qa_pipeline.doc_identity import DocsCanonicalDocIdResolver -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsDiagnostics, DocsEvidenceBundle, OpenAPIResult - - -class DocsDiagnosticsBuilder: - def __init__(self, resolver: DocsCanonicalDocIdResolver | None = None) -> None: - self._resolver = resolver or DocsCanonicalDocIdResolver() - - def build( - self, - *, - intent: str, - sub_intent: str, - planned_layers: list[str], - executed_layers: list[str], - non_empty_layers: list[str], - layer_diagnostics: dict[str, object], - evidence_bundle: DocsEvidenceBundle, - openapi_result: OpenAPIResult | None, - prompt_used: str, - llm_mode: str, - answer_mode: str, - output_valid: bool, - matched_intent_source: str, - matched_anchor_type: str, - matched_anchor_value: str | None, - exact_anchor_match: bool, - query_entity_candidates: list[str], - resolved_entity_candidates: list[str], - query_anchor_candidates: list[str], - resolved_anchor_candidates: list[str], - anchor_candidates: list[str], - selected_anchor: str | None, - anchor_selection_reason: str, - anchor_match_type: str, - docs_layers_with_hits: list[str], - gate_decision: str, - gate_decision_reason: str, - gate_missing_requirements: list[str], - gate_satisfied_requirements: list[str], - requested_fragment_type: str | None, - fragment_evidence_found: list[str], - fragment_missing_requirements: list[str], - prompt: dict[str, object], - degraded_reason: str | None, - fallback_used: bool, - code_intents_stubbed: bool, - ) -> DocsDiagnostics: - missing = list((openapi_result.diagnostics if openapi_result else {}).get("missing_required_fields") or []) - openapi_fields = 0 - if openapi_result is not None: - openapi_fields += len((openapi_result.request_schema or {}).get("properties") or {}) - openapi_fields += len((openapi_result.response_schema or {}).get("properties") or {}) - return DocsDiagnostics( - intent=intent, - sub_intent=sub_intent, - layers_used=list(planned_layers), - documents_found=len(evidence_bundle.documents), - facts_found=len(evidence_bundle.facts), - relations_found=len(evidence_bundle.relations), - openapi_fields_extracted=openapi_fields, - missing_required_fields=missing, - openapi_status=self._openapi_status(openapi_result), - prompt_used=prompt_used, - llm_mode=llm_mode, - output_valid=output_valid, - matched_intent_source=matched_intent_source, - matched_anchor_type=matched_anchor_type, - matched_anchor_value=matched_anchor_value, - exact_anchor_match=exact_anchor_match, - docs_layers_requested=list(planned_layers), - docs_layers_with_hits=list(docs_layers_with_hits), - planned_layers=list(planned_layers), - executed_layers=list(executed_layers), - non_empty_layers=list(non_empty_layers), - layer_diagnostics=dict(layer_diagnostics), - query_entity_candidates=list(query_entity_candidates), - resolved_entity_candidates=list(resolved_entity_candidates), - query_anchor_candidates=list(query_anchor_candidates), - resolved_anchor_candidates=list(resolved_anchor_candidates), - anchor_candidates=list(anchor_candidates), - selected_anchor=selected_anchor, - anchor_selection_reason=anchor_selection_reason, - anchor_match_type=anchor_match_type, - doc_ids=self._doc_ids(evidence_bundle, selected_anchor), - doc_paths=self._doc_paths(evidence_bundle), - doc_titles=self._doc_titles(evidence_bundle), - relation_hits_count=len(evidence_bundle.relations), - relation_targets=self._relation_targets(evidence_bundle), - selected_doc_ids=self._selected_doc_ids(evidence_bundle), - selected_fact_ids=self._selected_ids(evidence_bundle.facts, ("fact_id", "doc_id", "document_id"), fallback="path"), - selected_relation_ids=self._selected_ids( - evidence_bundle.relations, - ("relation_id", "target_doc_id", "target_document_id", "doc_id", "document_id"), - fallback="path", - ), - selected_chunk_ids=self._selected_ids(evidence_bundle.chunks, ("chunk_id", "doc_id", "document_id"), fallback="path"), - selected_entity_ids=self._selected_ids(evidence_bundle.entities, ("entity", "doc_id", "document_id"), fallback="title"), - selected_workflow_ids=self._selected_ids(evidence_bundle.workflows, ("workflow_id", "doc_id", "document_id"), fallback="path"), - fallback_doc_hits_count=len(evidence_bundle.documents) + len(evidence_bundle.chunks), - fallback_used=fallback_used, - fact_hits=len(evidence_bundle.facts), - entity_hits=self._entity_hits(evidence_bundle), - evidence_summary=self._evidence_summary(evidence_bundle, openapi_result), - gate_decision=gate_decision, - gate_decision_reason=gate_decision_reason, - gate_missing_requirements=list(gate_missing_requirements), - gate_satisfied_requirements=list(gate_satisfied_requirements), - openapi_evidence=dict(self._openapi_evidence(openapi_result)), - requested_fragment_type=requested_fragment_type, - fragment_evidence_found=list(fragment_evidence_found), - fragment_missing_requirements=list(fragment_missing_requirements), - prompt=dict(prompt), - answer_mode=answer_mode, - degrade_reason=degraded_reason, - degraded_reason=degraded_reason, - code_intents_stubbed=code_intents_stubbed, - ) - - def _openapi_status(self, openapi_result: OpenAPIResult | None) -> dict[str, bool]: - diagnostics = openapi_result.diagnostics if openapi_result else {} - return { - "has_path": bool(diagnostics.get("has_path")), - "has_method": bool(diagnostics.get("has_method")), - "has_request": bool(diagnostics.get("has_request")), - "has_response": bool(diagnostics.get("has_response")), - } - - def _doc_ids(self, evidence_bundle: DocsEvidenceBundle, selected_anchor: str | None) -> list[str]: - values = self._collect_doc_ids(evidence_bundle) - for item in evidence_bundle.relations: - metadata = dict(item.get("metadata") or {}) - target = metadata.get("target_doc_id") or metadata.get("target_document_id") - if target: - values.append(str(target)) - result = self._dedupe(values) - if selected_anchor and selected_anchor in result: - return [selected_anchor, *[item for item in result if item != selected_anchor]] - if selected_anchor: - return [selected_anchor, *result] - return result - - def _selected_doc_ids(self, evidence_bundle: DocsEvidenceBundle) -> list[str]: - values = self._selected_ids(evidence_bundle.documents, ("doc_id", "document_id"), fallback="path") - values.extend(self._selected_ids(evidence_bundle.entities, ("doc_id", "document_id"), fallback="path")) - values.extend(self._selected_ids(evidence_bundle.workflows, ("doc_id", "document_id"), fallback="path")) - return self._dedupe(values) - - def _doc_paths(self, evidence_bundle: DocsEvidenceBundle) -> list[str]: - return self._collect_distinct(evidence_bundle, "path") - - def _doc_titles(self, evidence_bundle: DocsEvidenceBundle) -> list[str]: - return self._collect_distinct(evidence_bundle, "title") - - def _relation_targets(self, evidence_bundle: DocsEvidenceBundle) -> list[str]: - values: list[str] = [] - for item in evidence_bundle.relations: - metadata = dict(item.get("metadata") or {}) - target = ( - metadata.get("target_doc_id") - or metadata.get("target_document_id") - or metadata.get("related_to") - or metadata.get("document_id") - or metadata.get("doc_id") - or item.get("title") - or item.get("path") - ) - if target: - values.append(str(target)) - return self._dedupe(values) - - def _selected_ids(self, items: list[dict], metadata_keys: tuple[str, ...], *, fallback: str) -> list[str]: - values: list[str] = [] - for item in items: - metadata = dict(item.get("metadata") or {}) - candidate = None - for key in metadata_keys: - candidate = metadata.get(key) - if candidate: - break - if not candidate: - candidate = item.get(fallback) - if candidate: - values.append(str(candidate)) - return self._dedupe(values) - - def _collect_doc_ids(self, evidence_bundle: DocsEvidenceBundle) -> list[str]: - values: list[str] = [] - for item in evidence_bundle.raw_rows: - values.extend(self._resolver.candidates(item)) - return self._dedupe(values) - - def _collect_distinct(self, evidence_bundle: DocsEvidenceBundle, key: str, *, fallback_key: str | None = None) -> list[str]: - values: list[str] = [] - for item in evidence_bundle.raw_rows: - metadata = dict(item.get("metadata") or {}) - candidate = metadata.get(key) if key == "doc_id" else item.get(key) - if not candidate and fallback_key: - candidate = item.get(fallback_key) - if candidate: - values.append(str(candidate)) - return self._dedupe(values) - - def _dedupe(self, values: list[str]) -> list[str]: - result: list[str] = [] - seen: set[str] = set() - for value in values: - normalized = value.strip() - if not normalized: - continue - if normalized in seen: - continue - seen.add(normalized) - result.append(normalized) - return result - - def _entity_hits(self, evidence_bundle: DocsEvidenceBundle) -> int: - values = self._collect_distinct(evidence_bundle, "entity") - return len(values) - - def _evidence_summary(self, evidence_bundle: DocsEvidenceBundle, openapi_result: OpenAPIResult | None) -> dict[str, object]: - return { - "documents": len(evidence_bundle.documents), - "facts": len(evidence_bundle.facts), - "entities": len(evidence_bundle.entities), - "workflows": len(evidence_bundle.workflows), - "relations": len(evidence_bundle.relations), - "chunks": len(evidence_bundle.chunks), - "selected_doc_ids": self._selected_doc_ids(evidence_bundle), - "selected_fact_ids": self._selected_ids(evidence_bundle.facts, ("fact_id", "doc_id", "document_id"), fallback="path"), - "selected_relation_ids": self._selected_ids( - evidence_bundle.relations, - ("relation_id", "target_doc_id", "target_document_id", "doc_id", "document_id"), - fallback="path", - ), - "selected_chunk_ids": self._selected_ids(evidence_bundle.chunks, ("chunk_id", "doc_id", "document_id"), fallback="path"), - "entity_hits": self._entity_hits(evidence_bundle), - "openapi_signals": self._openapi_evidence(openapi_result), - } - - def _openapi_evidence(self, openapi_result: OpenAPIResult | None) -> dict[str, bool]: - diagnostics = openapi_result.diagnostics if openapi_result else {} - return { - "path_found": bool(diagnostics.get("has_path")), - "method_found": bool(diagnostics.get("has_method")), - "operation_semantics_found": bool(diagnostics.get("operation_semantics_found")), - "request_payload_found": bool(diagnostics.get("request_payload_found")), - "request_schema": bool(diagnostics.get("has_request")), - "request_fields_found": bool(diagnostics.get("request_fields_found")), - "response_payload_found": bool(diagnostics.get("response_payload_found")), - "response_schema": bool(diagnostics.get("has_response")), - "response_fields_found": bool(diagnostics.get("response_fields_found")), - "status_codes": bool(diagnostics.get("status_codes_found")), - "content_type_found": bool(diagnostics.get("content_type_found")), - "examples_found": bool(diagnostics.get("examples_found")), - "payload_description": bool(diagnostics.get("payload_description_found")), - } diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/doc_identity.py b/src/app/modules/agent/runtime/docs_qa_pipeline/doc_identity.py deleted file mode 100644 index 1cd2e2e..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/doc_identity.py +++ /dev/null @@ -1,47 +0,0 @@ -from __future__ import annotations - -import re - - -class DocsCanonicalDocIdResolver: - _DOC_PATH_RE = re.compile(r"docs/(?:documentation/)?(?P
api|domain|logic|architecture)/(?P[^/]+)\.md$", re.IGNORECASE) - _TITLE_ENDPOINT_RE = re.compile(r"/([a-z0-9_{}-]+)", re.IGNORECASE) - - def candidates(self, row: dict) -> list[str]: - metadata = dict(row.get("metadata") or {}) - values: list[str] = [] - for candidate in ( - metadata.get("document_id"), - metadata.get("doc_id"), - self.from_path(str(row.get("path") or "")), - self.from_title(str(row.get("title") or ""), str(row.get("path") or "")), - ): - value = str(candidate or "").strip() - if value and value not in values: - values.append(value) - return values - - def from_path(self, path: str) -> str | None: - match = self._DOC_PATH_RE.search((path or "").strip()) - if match is None: - return None - section = match.group("section").lower() - name = match.group("name").lower().replace("-", "_") - if section == "api" and not name.endswith("_endpoint"): - name = f"{name}_endpoint" - if section == "domain" and name.endswith("_entity"): - name = name[: -len("_entity")] - return f"{section}.{name}" - - def from_title(self, title: str, path: str) -> str | None: - normalized = (title or "").strip().lower() - if "/health" in normalized: - return "api.health_endpoint" - if "/send" in normalized: - return "api.send_message_endpoint" - match = self._TITLE_ENDPOINT_RE.search(normalized) - section = "api" if "/api/" in (path or "").lower() else "" - if match and section: - slug = match.group(1).replace("-", "_") - return f"{section}.{slug}_endpoint" - return None diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/evidence_builder.py b/src/app/modules/agent/runtime/docs_qa_pipeline/evidence_builder.py deleted file mode 100644 index 9eb1f6f..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/evidence_builder.py +++ /dev/null @@ -1,91 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle - - -class DocsEvidenceBuilder: - _CAPS = { - "SYSTEM_FLOW_EXPLAIN": {"documents": 1, "facts": 1, "entities": 0, "workflows": 2, "relations": 2, "chunks": 2}, - "COMPONENT_EXPLAIN": {"documents": 1, "facts": 3, "entities": 0, "workflows": 0, "relations": 2, "chunks": 2}, - "API_METHOD_EXPLAIN": {"documents": 1, "facts": 3, "entities": 0, "workflows": 1, "relations": 0, "chunks": 2}, - "ENTITY_EXPLAIN": {"documents": 1, "facts": 1, "entities": 2, "workflows": 0, "relations": 2, "chunks": 1}, - "RELATED_DOCS_EXPLAIN": {"documents": 2, "facts": 0, "entities": 1, "workflows": 0, "relations": 4, "chunks": 1}, - "GENERIC_QA": {"documents": 2, "facts": 0, "entities": 0, "workflows": 0, "relations": 0, "chunks": 1}, - "OPENAPI_METHOD_GENERATE": {"documents": 1, "facts": 4, "entities": 0, "workflows": 0, "relations": 0, "chunks": 2}, - "OPENAPI_FRAGMENT_GENERATE": {"documents": 1, "facts": 4, "entities": 0, "workflows": 0, "relations": 0, "chunks": 2}, - } - _LAYER_TO_BUCKET = { - "D1_DOCUMENT_CATALOG": "documents", - "D2_FACT_INDEX": "facts", - "D3_ENTITY_CATALOG": "entities", - "D4_WORKFLOW_INDEX": "workflows", - "D5_RELATION_GRAPH": "relations", - "D0_DOC_CHUNKS": "chunks", - } - - def build(self, *, intent: str, sub_intent: str, raw_rows: list[dict]) -> DocsEvidenceBundle: - buckets = {name: [] for name in self._LAYER_TO_BUCKET.values()} - for row in raw_rows: - bucket = self._LAYER_TO_BUCKET.get(str(row.get("layer") or "")) - if bucket is not None: - buckets[bucket].append(self._normalize_row(row)) - support_paths = self._support_paths(sub_intent, buckets) - caps = self._CAPS.get(sub_intent, self._CAPS["GENERIC_QA"]) - return DocsEvidenceBundle( - intent=intent, - sub_intent=sub_intent, - documents=buckets["documents"][: caps["documents"]], - facts=buckets["facts"][: caps["facts"]], - entities=buckets["entities"][: caps["entities"]], - workflows=buckets["workflows"][: caps["workflows"]], - relations=buckets["relations"][: caps["relations"]], - chunks=self._select_chunks(buckets["chunks"], support_paths, caps["chunks"]), - raw_rows=list(raw_rows), - ) - - def _normalize_row(self, row: dict) -> dict: - return { - "layer": str(row.get("layer") or ""), - "path": str(row.get("path") or ""), - "title": str(row.get("title") or ""), - "content": str(row.get("content") or ""), - "metadata": dict(row.get("metadata") or {}), - } - - def _support_paths(self, sub_intent: str, buckets: dict[str, list[dict]]) -> list[str]: - ordered: list[dict] = [] - if sub_intent == "SYSTEM_FLOW_EXPLAIN": - ordered = [*buckets["workflows"], *buckets["relations"], *buckets["documents"]] - elif sub_intent == "COMPONENT_EXPLAIN": - ordered = [*buckets["facts"], *buckets["relations"], *buckets["documents"]] - elif sub_intent == "API_METHOD_EXPLAIN": - ordered = [*buckets["facts"], *buckets["workflows"], *buckets["documents"]] - elif sub_intent == "ENTITY_EXPLAIN": - ordered = [*buckets["entities"], *buckets["relations"], *buckets["documents"]] - elif sub_intent == "RELATED_DOCS_EXPLAIN": - ordered = [*buckets["relations"], *buckets["documents"], *buckets["entities"]] - elif sub_intent == "GENERIC_QA": - ordered = [*buckets["documents"], *buckets["chunks"]] - else: - ordered = [*buckets["facts"], *buckets["documents"]] - paths: list[str] = [] - for item in ordered: - path = str(item.get("path") or "").strip() - if path and path not in paths: - paths.append(path) - return paths[:4] - - def _select_chunks(self, chunks: list[dict], support_paths: list[str], limit: int) -> list[dict]: - if limit <= 0: - return [] - targeted = [item for item in chunks if str(item.get("path") or "") in support_paths] - selected = targeted[:limit] - if len(selected) >= limit: - return selected - for item in chunks: - if item in selected: - continue - selected.append(item) - if len(selected) >= limit: - break - return selected diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/exact_anchor_matcher.py b/src/app/modules/agent/runtime/docs_qa_pipeline/exact_anchor_matcher.py deleted file mode 100644 index 2a2ec82..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/exact_anchor_matcher.py +++ /dev/null @@ -1,117 +0,0 @@ -from __future__ import annotations - -import re - - -class DocsExactAnchorMatcher: - _TOKEN_RE = re.compile(r"[a-z0-9_./{}-]+", re.IGNORECASE) - _STRICT_ANCHOR_TYPES = {"endpoint", "entity"} - - def filter_rows(self, rows: list[dict], *, anchor_type: str, anchor_value: str | None) -> tuple[list[dict], bool]: - if anchor_type == "none" or not anchor_value or anchor_type not in self._STRICT_ANCHOR_TYPES: - return list(rows), False - exact = [row for row in rows if self._matches(row, anchor_type=anchor_type, anchor_value=anchor_value)] - if exact: - return self._expand_support_rows(rows, exact), True - return [], False - - def resolved_entity_candidates(self, rows: list[dict]) -> list[str]: - values: list[str] = [] - for row in rows: - metadata = dict(row.get("metadata") or {}) - for key in ("entity", "component"): - candidate = metadata.get(key) - if candidate: - values.append(str(candidate)) - return self._dedupe(values) - - def resolved_anchor_candidates(self, rows: list[dict]) -> list[str]: - values: list[str] = [] - for row in rows: - metadata = dict(row.get("metadata") or {}) - for candidate in ( - metadata.get("endpoint"), - metadata.get("document_id"), - metadata.get("doc_id"), - metadata.get("entity"), - metadata.get("component"), - row.get("path"), - ): - if candidate: - values.append(str(candidate)) - return self._dedupe(values) - - def _matches(self, row: dict, *, anchor_type: str, anchor_value: str) -> bool: - haystacks = [ - str(row.get("path") or "").lower(), - str(row.get("title") or "").lower(), - str(row.get("content") or "").lower(), - str(dict(row.get("metadata") or {}).get("endpoint") or "").lower(), - str(dict(row.get("metadata") or {}).get("entity") or "").lower(), - str(dict(row.get("metadata") or {}).get("component") or "").lower(), - str(dict(row.get("metadata") or {}).get("document_id") or "").lower(), - str(dict(row.get("metadata") or {}).get("doc_id") or "").lower(), - ] - needle = anchor_value.lower().strip() - if anchor_type == "endpoint": - return any(needle == value or f" {needle}" in f" {value}" for value in haystacks if value) - return any(needle in value for value in haystacks if value) - - def _expand_support_rows(self, rows: list[dict], exact: list[dict]) -> list[dict]: - doc_ids = self._doc_ids(exact) - paths = {str(row.get("path") or "").strip() for row in exact if str(row.get("path") or "").strip()} - expanded = list(exact) - for row in rows: - metadata = dict(row.get("metadata") or {}) - row_doc_ids = { - str(metadata.get("document_id") or "").strip(), - str(metadata.get("doc_id") or "").strip(), - } - row_path = str(row.get("path") or "").strip() - if row_path and row_path in paths: - expanded.append(row) - continue - if any(candidate and candidate in doc_ids for candidate in row_doc_ids): - expanded.append(row) - return self._dedupe_rows(expanded) - - def _doc_ids(self, rows: list[dict]) -> set[str]: - values: set[str] = set() - for row in rows: - metadata = dict(row.get("metadata") or {}) - for candidate in (metadata.get("document_id"), metadata.get("doc_id")): - value = str(candidate or "").strip() - if value: - values.add(value) - return values - - def _dedupe_rows(self, rows: list[dict]) -> list[dict]: - result: list[dict] = [] - seen: set[tuple[str, str, str, int | None, int | None]] = set() - for row in rows: - key = ( - str(row.get("layer") or ""), - str(row.get("path") or ""), - str(row.get("title") or ""), - row.get("span_start"), - row.get("span_end"), - ) - if key in seen: - continue - seen.add(key) - result.append(row) - return result - - def _dedupe(self, values: list[str]) -> list[str]: - result: list[str] = [] - seen: set[str] = set() - for value in values: - normalized = value.strip() - if not normalized: - continue - key = normalized.lower() - if key in seen: - continue - seen.add(key) - result.append(normalized) - return result diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/models.py b/src/app/modules/agent/runtime/docs_qa_pipeline/models.py deleted file mode 100644 index 43bf800..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/models.py +++ /dev/null @@ -1,114 +0,0 @@ -from __future__ import annotations - -from dataclasses import dataclass, field -from typing import Any - -from pydantic import BaseModel, ConfigDict, Field - - -class OpenAPIResult(BaseModel): - model_config = ConfigDict(extra="forbid") - - path: str = "" - method: str = "" - request_schema: dict[str, Any] | None = None - response_schema: dict[str, Any] | None = None - raw_yaml: str = "" - diagnostics: dict[str, Any] = Field(default_factory=dict) - - -class DocsDiagnostics(BaseModel): - model_config = ConfigDict(extra="forbid") - - intent: str - sub_intent: str - layers_used: list[str] = Field(default_factory=list) - documents_found: int = 0 - facts_found: int = 0 - relations_found: int = 0 - openapi_fields_extracted: int = 0 - missing_required_fields: list[str] = Field(default_factory=list) - openapi_status: dict[str, bool] = Field(default_factory=dict) - prompt_used: str = "" - llm_mode: str = "prose" - output_valid: bool = True - matched_intent_source: str = "deterministic" - matched_anchor_type: str = "none" - matched_anchor_value: str | None = None - exact_anchor_match: bool = False - docs_layers_requested: list[str] = Field(default_factory=list) - docs_layers_with_hits: list[str] = Field(default_factory=list) - planned_layers: list[str] = Field(default_factory=list) - executed_layers: list[str] = Field(default_factory=list) - non_empty_layers: list[str] = Field(default_factory=list) - layer_diagnostics: dict[str, Any] = Field(default_factory=dict) - query_entity_candidates: list[str] = Field(default_factory=list) - resolved_entity_candidates: list[str] = Field(default_factory=list) - query_anchor_candidates: list[str] = Field(default_factory=list) - resolved_anchor_candidates: list[str] = Field(default_factory=list) - anchor_candidates: list[str] = Field(default_factory=list) - selected_anchor: str | None = None - anchor_selection_reason: str = "" - anchor_match_type: str = "" - doc_ids: list[str] = Field(default_factory=list) - doc_paths: list[str] = Field(default_factory=list) - doc_titles: list[str] = Field(default_factory=list) - relation_hits_count: int = 0 - relation_targets: list[str] = Field(default_factory=list) - selected_doc_ids: list[str] = Field(default_factory=list) - selected_fact_ids: list[str] = Field(default_factory=list) - selected_relation_ids: list[str] = Field(default_factory=list) - selected_chunk_ids: list[str] = Field(default_factory=list) - selected_entity_ids: list[str] = Field(default_factory=list) - selected_workflow_ids: list[str] = Field(default_factory=list) - fallback_doc_hits_count: int = 0 - fallback_used: bool = False - fact_hits: int = 0 - entity_hits: int = 0 - evidence_summary: dict[str, Any] = Field(default_factory=dict) - gate_decision: str = "ready" - gate_decision_reason: str = "" - gate_missing_requirements: list[str] = Field(default_factory=list) - gate_satisfied_requirements: list[str] = Field(default_factory=list) - openapi_evidence: dict[str, bool] = Field(default_factory=dict) - requested_fragment_type: str | None = None - fragment_evidence_found: list[str] = Field(default_factory=list) - fragment_missing_requirements: list[str] = Field(default_factory=list) - prompt: dict[str, Any] = Field(default_factory=dict) - answer_mode: str = "answered" - degrade_reason: str | None = None - degraded_reason: str | None = None - code_intents_stubbed: bool = False - - -@dataclass(slots=True) -class DocsEvidenceBundle: - intent: str - sub_intent: str - documents: list[dict[str, Any]] = field(default_factory=list) - facts: list[dict[str, Any]] = field(default_factory=list) - entities: list[dict[str, Any]] = field(default_factory=list) - workflows: list[dict[str, Any]] = field(default_factory=list) - relations: list[dict[str, Any]] = field(default_factory=list) - chunks: list[dict[str, Any]] = field(default_factory=list) - raw_rows: list[dict[str, Any]] = field(default_factory=list) - - -@dataclass(slots=True) -class DocsQAPipelineResult: - user_query: str - rag_session_id: str - router_result: Any - retrieval_request: Any - evidence_bundle: DocsEvidenceBundle - answer: str - diagnostics: DocsDiagnostics - openapi_result: OpenAPIResult | None = None - prompt_name: str = "" - llm_request: dict[str, Any] = field(default_factory=dict) - output_valid: bool = True - answer_mode: str = "answered" - degraded_reason: str = "" - raw_rows: list[dict[str, Any]] = field(default_factory=list) - timings_ms: dict[str, int] = field(default_factory=dict) - mode: str = "full" diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_evidence_extractor.py b/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_evidence_extractor.py deleted file mode 100644 index 62f3358..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_evidence_extractor.py +++ /dev/null @@ -1,187 +0,0 @@ -from __future__ import annotations - -import re -from typing import Any - -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle - - -class OpenAPIEvidenceExtractor: - _PATH_RE = re.compile(r"(/[a-z0-9_./{}-]+)", re.IGNORECASE) - _METHOD_RE = re.compile(r"\b(get|post|put|patch|delete)\b", re.IGNORECASE) - _FIELD_RE = re.compile(r"\b([a-z_][a-z0-9_]{1,40})\b") - _STATUS_RE = re.compile(r"\b(200|201|202|204|400|401|403|404|409|422|500|503)\b") - _FIELD_STOPWORDS = { - "request", - "response", - "schema", - "payload", - "fields", - "contains", - "type", - "properties", - "post", - "get", - "put", - "patch", - "delete", - "endpoint", - "returns", - "return", - "with", - "and", - "for", - "send", - "health", - "status", - "result", - "body", - } - - def extract(self, bundle: DocsEvidenceBundle, *, requested_fragment_type: str | None) -> dict[str, Any]: - items = [*bundle.facts, *bundle.documents, *bundle.chunks] - path = self._extract_path(items) - method = self._extract_method(items) - request_schema = self._extract_schema(items, "request") - response_schema = self._extract_schema(items, "response") - if request_schema is None and requested_fragment_type == "request_schema": - request_schema = self._schema_from_text(items, kind="request") - if response_schema is None and requested_fragment_type == "response_schema": - response_schema = self._schema_from_text(items, kind="response") - if request_schema is None and response_schema is None and requested_fragment_type == "schema_fragment": - request_schema = self._schema_from_text(items, kind="schema") - status_codes = self._status_codes(items) - content_type = self._content_type(items) - operation_summary = self._operation_summary(items) - examples_found = self._examples_found(items) - diagnostics = { - "has_path": bool(path), - "has_method": bool(method), - "has_request": request_schema is not None, - "has_response": response_schema is not None, - "operation_semantics_found": bool(operation_summary), - "request_payload_found": request_schema is not None or self._has_payload_text(items, "request"), - "request_fields_found": self._field_count(request_schema) > 0, - "response_payload_found": response_schema is not None or self._has_payload_text(items, "response"), - "response_fields_found": self._field_count(response_schema) > 0, - "status_codes_found": bool(status_codes), - "content_type_found": bool(content_type), - "examples_found": examples_found, - "payload_description_found": self._has_payload_description(items), - "status_codes": status_codes, - "content_type": content_type, - "operation_summary": operation_summary, - } - return { - "path": path, - "method": method, - "request_schema": request_schema, - "response_schema": response_schema, - "diagnostics": diagnostics, - } - - def _extract_path(self, items: list[dict[str, Any]]) -> str: - for item in items: - metadata = dict(item.get("metadata") or {}) - if metadata.get("endpoint"): - return str(metadata["endpoint"]) - for source in (item.get("title"), item.get("content")): - match = self._PATH_RE.search(str(source or "")) - if match: - return match.group(1) - return "" - - def _extract_method(self, items: list[dict[str, Any]]) -> str: - for item in items: - metadata = dict(item.get("metadata") or {}) - if metadata.get("http_method"): - return str(metadata["http_method"]).lower() - for source in (item.get("title"), item.get("content")): - match = self._METHOD_RE.search(str(source or "")) - if match: - return match.group(1).lower() - return "" - - def _extract_schema(self, items: list[dict[str, Any]], kind: str) -> dict[str, Any] | None: - for item in items: - metadata = dict(item.get("metadata") or {}) - candidate = metadata.get(f"{kind}_schema") or metadata.get(f"{kind}_fields") - schema = self._as_schema(candidate) - if schema is not None: - return schema - return None - - def _schema_from_text(self, items: list[dict[str, Any]], *, kind: str) -> dict[str, Any] | None: - markers = { - "request": ("request", "payload", "body", "fields", "message", "chat_id"), - "response": ("response", "returns", "result", "status", "body"), - "schema": ("schema", "payload", "fields", "properties"), - } - fields: list[str] = [] - for item in items: - text = " ".join((str(item.get("title") or ""), str(item.get("content") or ""))).lower() - if not any(marker in text for marker in markers[kind]): - continue - for token in self._FIELD_RE.findall(text): - if token in self._FIELD_STOPWORDS: - continue - if token not in fields: - fields.append(token) - if not fields: - return None - properties = {name: {"type": "string"} for name in fields[:8]} - required = [name for name in fields[:3] if len(name) > 1] - result: dict[str, Any] = {"type": "object", "properties": properties} - if required: - result["required"] = required - return result - - def _as_schema(self, value: Any) -> dict[str, Any] | None: - if isinstance(value, dict): - return value - if isinstance(value, list): - properties = {str(item): {"type": "string"} for item in value if str(item).strip()} - return {"type": "object", "properties": properties} if properties else None - return None - - def _status_codes(self, items: list[dict[str, Any]]) -> list[str]: - values: list[str] = [] - for item in items: - text = " ".join((str(item.get("title") or ""), str(item.get("content") or ""))) - for code in self._STATUS_RE.findall(text): - if code not in values: - values.append(code) - return values - - def _content_type(self, items: list[dict[str, Any]]) -> str: - for item in items: - text = " ".join((str(item.get("title") or ""), str(item.get("content") or ""))).lower() - for marker in ("application/json", "json", "multipart/form-data", "x-www-form-urlencoded"): - if marker in text: - return marker - return "" - - def _operation_summary(self, items: list[dict[str, Any]]) -> str: - for item in items: - for source in (item.get("title"), item.get("content")): - text = str(source or "").strip() - if len(text) >= 8: - return text[:120] - return "" - - def _examples_found(self, items: list[dict[str, Any]]) -> bool: - for item in items: - text = " ".join((str(item.get("title") or ""), str(item.get("content") or ""))).lower() - if any(marker in text for marker in ("example", "пример", "{", "}")): - return True - return False - - def _has_payload_text(self, items: list[dict[str, Any]], kind: str) -> bool: - markers = ("payload", "body", "fields", "request") if kind == "request" else ("response", "returns", "status", "body") - return any(any(marker in str(item.get("content") or "").lower() for marker in markers) for item in items) - - def _has_payload_description(self, items: list[dict[str, Any]]) -> bool: - return any(len(str(item.get("content") or "").strip()) >= 12 for item in items) - - def _field_count(self, schema: dict[str, Any] | None) -> int: - return len((schema or {}).get("properties") or {}) diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_generator.py b/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_generator.py deleted file mode 100644 index 211866f..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_generator.py +++ /dev/null @@ -1,161 +0,0 @@ -from __future__ import annotations - -from typing import Any - -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle, OpenAPIResult -from app.modules.agent.runtime.docs_qa_pipeline.openapi_evidence_extractor import OpenAPIEvidenceExtractor - - -class OpenAPIGenerator: - def __init__(self, extractor: OpenAPIEvidenceExtractor | None = None) -> None: - self._extractor = extractor or OpenAPIEvidenceExtractor() - - def generate( - self, - evidence_bundle: DocsEvidenceBundle, - mode: str, - *, - requested_fragment_type: str | None = None, - ) -> OpenAPIResult: - extracted = self._extractor.extract(evidence_bundle, requested_fragment_type=requested_fragment_type) - path = str(extracted["path"] or "") - method = str(extracted["method"] or "") - request_schema = extracted["request_schema"] - response_schema = extracted["response_schema"] - diagnostics = dict(extracted["diagnostics"] or {}) - diagnostics["missing_required_fields"] = self._missing_fields( - path=path, - method=method, - request_schema=request_schema, - response_schema=response_schema, - mode=mode, - requested_fragment_type=requested_fragment_type, - diagnostics=diagnostics, - ) - raw_yaml = self._render( - path=path, - method=method, - request_schema=request_schema, - response_schema=response_schema, - mode=mode, - requested_fragment_type=requested_fragment_type, - diagnostics=diagnostics, - ) - return OpenAPIResult( - path=path, - method=method, - request_schema=request_schema, - response_schema=response_schema, - raw_yaml=raw_yaml, - diagnostics=diagnostics, - ) - - def _missing_fields( - self, - *, - path: str, - method: str, - request_schema: dict[str, Any] | None, - response_schema: dict[str, Any] | None, - mode: str, - requested_fragment_type: str | None, - diagnostics: dict[str, Any], - ) -> list[str]: - missing: list[str] = [] - if not path: - missing.append("path") - if mode == "OPENAPI_FRAGMENT_GENERATE": - if requested_fragment_type == "request_schema" and request_schema is None and not diagnostics.get("request_payload_found"): - missing.append("request_schema") - elif requested_fragment_type == "response_schema" and response_schema is None and not diagnostics.get("response_payload_found"): - missing.append("response_schema") - elif requested_fragment_type == "parameters" and not diagnostics.get("has_method"): - missing.append("parameters") - elif request_schema is None and response_schema is None and not diagnostics.get("payload_description_found"): - missing.append("schema_fragment") - return missing - if not method: - missing.append("method") - if request_schema is None and not diagnostics.get("request_payload_found"): - missing.append("request_schema") - if response_schema is None and not diagnostics.get("response_payload_found") and not diagnostics.get("status_codes_found"): - missing.append("response_schema") - return missing - - def _render( - self, - *, - path: str, - method: str, - request_schema: dict[str, Any] | None, - response_schema: dict[str, Any] | None, - mode: str, - requested_fragment_type: str | None, - diagnostics: dict[str, Any], - ) -> str: - if mode == "OPENAPI_FRAGMENT_GENERATE": - if requested_fragment_type == "response_schema": - return self._render_schema(response_schema, diagnostics, "Documented response fragment") - return self._render_schema(request_schema or response_schema, diagnostics, "Documented schema fragment") - if not path: - return "" - method_line = method or "get" - summary = str(diagnostics.get("operation_summary") or "Documented API method") - response_block = self._render_responses(response_schema, diagnostics) - request_block = self._render_request_body(request_schema, diagnostics) - return "\n".join( - [ - "paths:", - f" {path}:", - f" {method_line}:", - f" summary: \"{summary}\"", - *request_block, - " responses:", - *response_block, - ] - ) - - def _render_schema(self, schema: dict[str, Any] | None, diagnostics: dict[str, Any], fallback_description: str) -> str: - if schema is None: - return "\n".join(["type: object", f"description: \"{fallback_description}\""]) - return self._yaml_from_object(schema, indent=0) - - def _render_request_body(self, schema: dict[str, Any] | None, diagnostics: dict[str, Any]) -> list[str]: - lines = [" requestBody:"] - if schema is None: - description = "Documented request payload" if diagnostics.get("request_payload_found") else "Request payload not fully documented" - lines.append(f" description: \"{description}\"") - return lines - lines.append(" content:") - content_type = str(diagnostics.get("content_type") or "application/json") - lines.append(f" {content_type}:") - lines.append(" schema:") - lines.extend(self._yaml_from_object(schema, indent=14).splitlines()) - return lines - - def _render_responses(self, schema: dict[str, Any] | None, diagnostics: dict[str, Any]) -> list[str]: - status_codes = list(diagnostics.get("status_codes") or []) or ["200"] - code = status_codes[0] - lines = [f" \"{code}\":", " description: \"Documented response\""] - if schema is not None: - lines.append(" content:") - content_type = str(diagnostics.get("content_type") or "application/json") - lines.append(f" {content_type}:") - lines.append(" schema:") - lines.extend(self._yaml_from_object(schema, indent=16).splitlines()) - return lines - - def _yaml_from_object(self, value: dict[str, Any], *, indent: int) -> str: - lines: list[str] = [] - prefix = " " * indent - for key, item in value.items(): - if isinstance(item, dict): - lines.append(f"{prefix}{key}:") - lines.append(self._yaml_from_object(item, indent=indent + 2)) - elif isinstance(item, list): - lines.append(f"{prefix}{key}:") - for row in item: - lines.append(f"{prefix} - {row}") - else: - lines.append(f"{prefix}{key}: {item}") - return "\n".join(lines) diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_postprocessor.py b/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_postprocessor.py deleted file mode 100644 index df94e31..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/openapi_postprocessor.py +++ /dev/null @@ -1,37 +0,0 @@ -from __future__ import annotations - -import yaml - - -class OpenAPIPostprocessor: - _ALLOWED_METHODS = {"get", "post", "put", "delete", "patch"} - - def validate(self, answer: str, *, require_paths: bool) -> tuple[bool, dict]: - try: - payload = yaml.safe_load(answer) or {} - except yaml.YAMLError: - return False, {"reason": "invalid_yaml"} - if not isinstance(payload, dict): - return False, {"reason": "invalid_yaml_root"} - if require_paths: - paths = payload.get("paths") - if not isinstance(paths, dict) or not paths: - return False, {"reason": "missing_paths"} - methods = self._methods(paths) - if not methods: - return False, {"reason": "missing_method"} - return True, {"reason": "ok", "methods": methods} - if not isinstance(payload, dict) or not payload: - return False, {"reason": "empty_schema"} - return True, {"reason": "ok"} - - def _methods(self, paths: dict) -> list[str]: - methods: list[str] = [] - for item in paths.values(): - if not isinstance(item, dict): - continue - for method in item.keys(): - method_name = str(method).lower() - if method_name in self._ALLOWED_METHODS and method_name not in methods: - methods.append(method_name) - return methods diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/pipeline.py b/src/app/modules/agent/runtime/docs_qa_pipeline/pipeline.py deleted file mode 100644 index 7755a3f..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/pipeline.py +++ /dev/null @@ -1,702 +0,0 @@ -from __future__ import annotations - -from time import perf_counter -from typing import Any - -from app.modules.agent.llm import AgentLlmService -from app.modules.agent.intent_router_v2.analysis.docs_query_signals import DocsQuerySignals -from app.modules.agent.runtime.docs_qa_pipeline.answer_synthesizer import DocsAnswerSynthesizer -from app.modules.agent.runtime.docs_qa_pipeline.anchor_selector import DocsAnchorSelector -from app.modules.agent.runtime.docs_qa_pipeline.diagnostics_builder import DocsDiagnosticsBuilder -from app.modules.agent.runtime.docs_qa_pipeline.evidence_builder import DocsEvidenceBuilder -from app.modules.agent.runtime.docs_qa_pipeline.exact_anchor_matcher import DocsExactAnchorMatcher -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsQAPipelineResult -from app.modules.agent.runtime.docs_qa_pipeline.openapi_postprocessor import OpenAPIPostprocessor -from app.modules.agent.runtime.docs_qa_pipeline.openapi_generator import OpenAPIGenerator -from app.modules.agent.runtime.docs_qa_pipeline.prompt_payload_builder import DocsPromptPayloadBuilder -from app.modules.agent.runtime.legacy_pipeline import RetrievalAdapter -from app.modules.agent.runtime.steps.context import build_retrieval_request -from app.modules.agent.runtime.steps.generation import RuntimePromptSelector -from app.modules.agent.observability.module_trace import RequestTraceContext - - -class DocsQAPipelineRunner: - def __init__( - self, - router: Any, - retrieval_adapter: RetrievalAdapter, - repo_context: Any = None, - llm: AgentLlmService | None = None, - evidence_builder: DocsEvidenceBuilder | None = None, - answer_synthesizer: DocsAnswerSynthesizer | None = None, - openapi_generator: OpenAPIGenerator | None = None, - diagnostics_builder: DocsDiagnosticsBuilder | None = None, - prompt_selector: RuntimePromptSelector | None = None, - prompt_payload_builder: DocsPromptPayloadBuilder | None = None, - openapi_postprocessor: OpenAPIPostprocessor | None = None, - exact_anchor_matcher: DocsExactAnchorMatcher | None = None, - anchor_selector: DocsAnchorSelector | None = None, - ) -> None: - self._router = router - self._adapter = retrieval_adapter - self._repo_context = repo_context - self._llm = llm - self._evidence_builder = evidence_builder or DocsEvidenceBuilder() - self._answer_synthesizer = answer_synthesizer or DocsAnswerSynthesizer() - self._openapi_generator = openapi_generator or OpenAPIGenerator() - self._diagnostics_builder = diagnostics_builder or DocsDiagnosticsBuilder() - self._prompt_selector = prompt_selector or RuntimePromptSelector() - self._prompt_payload_builder = prompt_payload_builder or DocsPromptPayloadBuilder() - self._openapi_postprocessor = openapi_postprocessor or OpenAPIPostprocessor() - self._exact_anchor_matcher = exact_anchor_matcher or DocsExactAnchorMatcher() - self._anchor_selector = anchor_selector or DocsAnchorSelector() - self._docs_signals = DocsQuerySignals() - - def run( - self, - user_query: str, - rag_session_id: str, - *, - conversation_state: Any = None, - mode: str = "full", - trace: RequestTraceContext | None = None, - ) -> DocsQAPipelineResult: - timings: dict[str, int] = {} - t0 = perf_counter() - router_result = self._router.route( - user_query, - conversation_state or _default_conversation_state(), - self._repo_context or _default_repo_context(), - ) - timings["router"] = _ms(t0) - - t1 = perf_counter() - request = build_retrieval_request(router_result, rag_session_id) - raw_rows = self._adapter.retrieve_with_plan( - rag_session_id, - request.query, - request.retrieval_spec, - request.retrieval_constraints, - query_plan=request.query_plan, - ) - retrieval_report = self._adapter.consume_retrieval_report() if hasattr(self._adapter, "consume_retrieval_report") else {} - unfiltered_rows = list(raw_rows) - raw_rows, exact_anchor_match = self._exact_anchor_matcher.filter_rows( - raw_rows, - anchor_type=router_result.matched_anchor_type, - anchor_value=router_result.matched_anchor_value, - ) - if request.sub_intent == "RELATED_DOCS_EXPLAIN" and not raw_rows and self._has_relation_hits(unfiltered_rows): - raw_rows = unfiltered_rows - if trace is not None: - trace.module("rag_retrieval").log( - "completed", - { - "planned_layers": list(request.requested_layers), - "executed_layers": list(retrieval_report.get("executed_layers") or request.requested_layers), - "layer_diagnostics": dict(retrieval_report.get("layer_diagnostics") or {}), - "rows": len(raw_rows), - }, - ) - timings["retrieval"] = _ms(t1) - - t2 = perf_counter() - evidence_bundle = self._evidence_builder.build( - intent=router_result.intent, - sub_intent=request.sub_intent, - raw_rows=raw_rows, - ) - anchor_selection = self._anchor_selector.select( - sub_intent=request.sub_intent, - anchor_type=router_result.matched_anchor_type, - anchor_value=router_result.matched_anchor_value, - rows=raw_rows, - ) - openapi_result = None - prompt_name = self._prompt_selector.select( - intent=router_result.intent, - sub_intent=request.sub_intent, - answer_mode="normal", - ) - llm_mode = self._llm_mode(router_result.intent, request.sub_intent) - output_valid = True - answer_mode = "diagnostic_only" if mode == "pre_llm_only" else "answered" - degraded_reason = "" - answer = "" - llm_request: dict[str, Any] = {} - requested_fragment_type = self._requested_fragment_type(user_query, request.sub_intent) - if router_result.intent in {"OPENAPI_GENERATION", "OPENAPI_FROM_DOCUMENTATION"}: - openapi_result = self._openapi_generator.generate( - evidence_bundle, - request.sub_intent, - requested_fragment_type=requested_fragment_type, - ) - llm_request = self._build_llm_request( - question=user_query, - intent=router_result.intent, - sub_intent=request.sub_intent, - evidence_bundle=evidence_bundle, - prompt_name=prompt_name, - log_context="graph.project_qa.docs.openapi", - api_contract=openapi_result, - ) - if mode == "pre_llm_only": - answer_mode, degraded_reason, output_valid = self._evaluate_openapi_gate( - user_query=user_query, - sub_intent=request.sub_intent, - router_result=router_result, - openapi_result=openapi_result, - exact_anchor_match=exact_anchor_match, - ) - answer = openapi_result.raw_yaml if answer_mode != "degraded" else "Недостаточно contract evidence для OpenAPI." - else: - answer = self._generate_openapi_answer( - user_query, - router_result.intent, - request.sub_intent, - evidence_bundle, - openapi_result, - trace=trace, - ) - output_valid, llm_details = self._openapi_postprocessor.validate( - answer, - require_paths=request.sub_intent != "OPENAPI_FRAGMENT_GENERATE", - ) - if not output_valid: - answer = openapi_result.raw_yaml - openapi_result = openapi_result.model_copy( - update={"diagnostics": {**openapi_result.diagnostics, "llm_validation": llm_details}} - ) - answer_mode, degraded_reason, output_valid, answer = self._finalize_openapi_answer( - answer=answer, - router_result=router_result, - openapi_result=openapi_result, - output_valid=output_valid, - exact_anchor_match=exact_anchor_match, - ) - else: - llm_request = self._build_llm_request( - question=user_query, - intent=router_result.intent, - sub_intent=request.sub_intent, - evidence_bundle=evidence_bundle, - prompt_name=prompt_name, - log_context="graph.project_qa.docs.answer", - ) - if mode == "pre_llm_only": - answer_mode, degraded_reason = self._evaluate_docs_gate( - raw_rows=raw_rows, - sub_intent=request.sub_intent, - matched_anchor_type=router_result.matched_anchor_type, - exact_anchor_match=exact_anchor_match, - matched_anchor_value=router_result.matched_anchor_value, - ) - output_valid = answer_mode != "degraded" - else: - answer = self._generate_docs_answer( - user_query, - router_result.intent, - request.sub_intent, - evidence_bundle, - trace=trace, - ) - answer_mode, degraded_reason, answer = self._finalize_docs_answer( - answer=answer, - raw_rows=raw_rows, - sub_intent=request.sub_intent, - matched_anchor_type=router_result.matched_anchor_type, - exact_anchor_match=exact_anchor_match, - matched_anchor_value=router_result.matched_anchor_value, - ) - gate_decision, gate_decision_reason, gate_missing_requirements, gate_satisfied_requirements = self._build_gate_details( - intent=router_result.intent, - sub_intent=request.sub_intent, - answer_mode=answer_mode, - degraded_reason=degraded_reason, - raw_rows=raw_rows, - exact_anchor_match=exact_anchor_match, - matched_anchor_type=router_result.matched_anchor_type, - matched_anchor_value=router_result.matched_anchor_value, - openapi_result=openapi_result, - router_result=router_result, - ) - if trace is not None: - trace.module("evidence_gate").log( - "evaluated", - { - "decision": gate_decision, - "reason": gate_decision_reason, - "missing": gate_missing_requirements, - "satisfied": gate_satisfied_requirements, - }, - ) - diagnostics = self._diagnostics_builder.build( - intent=router_result.intent, - sub_intent=request.sub_intent, - planned_layers=list(request.requested_layers), - executed_layers=list(retrieval_report.get("executed_layers") or request.requested_layers), - non_empty_layers=self._non_empty_layers(raw_rows, retrieval_report, request.sub_intent), - layer_diagnostics=dict(retrieval_report.get("layer_diagnostics") or {}), - evidence_bundle=evidence_bundle, - openapi_result=openapi_result, - prompt_used=prompt_name, - llm_mode=llm_mode, - answer_mode=answer_mode, - output_valid=output_valid, - matched_intent_source=router_result.matched_intent_source, - matched_anchor_type=router_result.matched_anchor_type, - matched_anchor_value=router_result.matched_anchor_value, - exact_anchor_match=exact_anchor_match, - query_entity_candidates=self._docs_signals.query_entity_candidates(user_query), - resolved_entity_candidates=self._exact_anchor_matcher.resolved_entity_candidates(raw_rows), - query_anchor_candidates=self._docs_signals.query_anchor_candidates(user_query), - resolved_anchor_candidates=self._exact_anchor_matcher.resolved_anchor_candidates(raw_rows), - anchor_candidates=list(anchor_selection.get("anchor_candidates") or []), - selected_anchor=anchor_selection.get("selected_anchor"), - anchor_selection_reason=str(anchor_selection.get("anchor_selection_reason") or ""), - anchor_match_type=str(anchor_selection.get("anchor_match_type") or ""), - docs_layers_with_hits=self._non_empty_layers(raw_rows, retrieval_report, request.sub_intent), - gate_decision=gate_decision, - gate_decision_reason=gate_decision_reason, - gate_missing_requirements=gate_missing_requirements, - gate_satisfied_requirements=gate_satisfied_requirements, - requested_fragment_type=("method" if request.sub_intent == "OPENAPI_METHOD_GENERATE" else requested_fragment_type), - fragment_evidence_found=self._fragment_evidence_found(requested_fragment_type, openapi_result), - fragment_missing_requirements=self._fragment_missing_requirements(requested_fragment_type, openapi_result), - prompt=llm_request, - degraded_reason=degraded_reason or None, - fallback_used=bool(dict(retrieval_report.get("fallback") or {}).get("used")), - code_intents_stubbed=False, - ) - timings["execution"] = _ms(t2) - return DocsQAPipelineResult( - user_query=user_query, - rag_session_id=rag_session_id, - router_result=router_result, - retrieval_request=request, - evidence_bundle=evidence_bundle, - answer=answer, - diagnostics=diagnostics, - openapi_result=openapi_result, - prompt_name=prompt_name, - llm_request=llm_request, - output_valid=output_valid, - answer_mode=answer_mode, - degraded_reason=degraded_reason, - raw_rows=raw_rows, - timings_ms=timings, - mode=mode, - ) - - def _generate_docs_answer(self, question: str, intent: str, sub_intent: str, evidence_bundle, trace=None) -> str: - if self._llm is None: - return self._answer_synthesizer.synthesize(question, evidence_bundle) - payload = self._prompt_payload_builder.build( - question=question, - intent=intent, - sub_intent=sub_intent, - evidence_bundle=evidence_bundle, - ) - prompt_name = self._prompt_selector.select(intent=intent, sub_intent=sub_intent, answer_mode="normal") - return self._llm.generate( - prompt_name, - payload, - log_context="graph.project_qa.docs.answer", - trace=trace.module("llm") if trace is not None else None, - ).strip() - - def _generate_openapi_answer(self, question: str, intent: str, sub_intent: str, evidence_bundle, api_contract, trace=None) -> str: - if self._llm is None: - return api_contract.raw_yaml - payload = self._prompt_payload_builder.build( - question=question, - intent=intent, - sub_intent=sub_intent, - evidence_bundle=evidence_bundle, - api_contract=api_contract, - ) - prompt_name = self._prompt_selector.select(intent=intent, sub_intent=sub_intent, answer_mode="normal") - return self._llm.generate( - prompt_name, - payload, - log_context="graph.project_qa.docs.openapi", - trace=trace.module("llm") if trace is not None else None, - ).strip() - - def _llm_mode(self, intent: str, sub_intent: str) -> str: - if sub_intent == "RELATED_DOCS_EXPLAIN": - return "graph_summary" - if intent == "GENERAL_QA": - return "prose" - if intent in {"OPENAPI_GENERATION", "OPENAPI_FROM_DOCUMENTATION"}: - return "yaml" - return "prose" - - def _build_llm_request( - self, - *, - question: str, - intent: str, - sub_intent: str, - evidence_bundle, - prompt_name: str, - log_context: str, - api_contract=None, - ) -> dict[str, Any]: - user_prompt = self._prompt_payload_builder.build( - question=question, - intent=intent, - sub_intent=sub_intent, - evidence_bundle=evidence_bundle, - api_contract=api_contract, - ) - if self._llm is None: - return { - "prompt_name": prompt_name, - "system_prompt": "You are a helpful assistant.", - "user_prompt": user_prompt, - "log_context": log_context, - } - return self._llm.build_request(prompt_name, user_prompt, log_context=log_context) - - def _finalize_docs_answer( - self, - *, - answer: str, - raw_rows: list[dict], - sub_intent: str, - matched_anchor_type: str, - exact_anchor_match: bool, - matched_anchor_value: str | None, - ) -> tuple[str, str, str]: - if self._should_reject_on_exact_anchor( - sub_intent=sub_intent, - raw_rows=raw_rows, - matched_anchor_type=matched_anchor_type, - matched_anchor_value=matched_anchor_value, - exact_anchor_match=exact_anchor_match, - ): - return "degraded", "not_found_exact_anchor", "Не найдено точное совпадение по запрошенному docs anchor." - if not raw_rows: - return "degraded", "retrieval_empty", "По документации не найдено релевантных данных." - if not answer.strip(): - return "degraded", "insufficient_docs_evidence", "Недостаточно подтвержденных данных в документации." - return "answered", "", answer - - def _finalize_openapi_answer( - self, - *, - answer: str, - router_result, - openapi_result, - output_valid: bool, - exact_anchor_match: bool, - ) -> tuple[str, str, bool, str]: - requested_endpoint = str(router_result.matched_anchor_value or "").strip() - generated_endpoint = str(openapi_result.path or "").strip() - diagnostics = dict(openapi_result.diagnostics or {}) - if requested_endpoint and not exact_anchor_match: - return "degraded", "not_found_exact_anchor", False, "Не найден точный endpoint в документации." - if requested_endpoint and generated_endpoint and requested_endpoint != generated_endpoint: - return "degraded", "generated_endpoint_mismatch", False, "OpenAPI сгенерирован не по запрошенному endpoint." - if str(getattr(router_result.retrieval_spec.filters, "doc_type", "") or "") != "api_method": - return "degraded", "wrong_doc_type_match", False, "Для OpenAPI не найден подходящий api_method evidence." - if not generated_endpoint: - return "degraded", "insufficient_docs_evidence", False, "Недостаточно contract evidence для OpenAPI." - has_partial_contract = any( - ( - openapi_result.request_schema is not None, - openapi_result.response_schema is not None, - diagnostics.get("status_codes_found"), - diagnostics.get("payload_description_found"), - ) - ) - if not has_partial_contract: - return "degraded", "insufficient_docs_evidence", False, "Недостаточно contract evidence для OpenAPI." - if not output_valid: - return "structured_spec_partial", "invalid_llm_output_fallback", False, answer - if openapi_result.method and openapi_result.response_schema is not None and openapi_result.request_schema is not None: - return "structured_spec", "", True, answer - if openapi_result.method or diagnostics.get("status_codes_found") or diagnostics.get("payload_description_found"): - return "structured_spec_partial", "answered_with_gaps", True, answer - return "degraded", "insufficient_docs_evidence", False, "Недостаточно contract evidence для OpenAPI." - - def _evaluate_docs_gate( - self, - *, - raw_rows: list[dict], - sub_intent: str, - matched_anchor_type: str, - exact_anchor_match: bool, - matched_anchor_value: str | None, - ) -> tuple[str, str]: - if self._should_reject_on_exact_anchor( - sub_intent=sub_intent, - raw_rows=raw_rows, - matched_anchor_type=matched_anchor_type, - matched_anchor_value=matched_anchor_value, - exact_anchor_match=exact_anchor_match, - ): - return "degraded", "not_found_exact_anchor" - if not raw_rows: - return "degraded", "retrieval_empty" - return "ready", "" - - def _evaluate_openapi_gate( - self, - *, - user_query: str, - sub_intent: str, - router_result, - openapi_result, - exact_anchor_match: bool, - ) -> tuple[str, str, bool]: - requested_endpoint = str(router_result.matched_anchor_value or "").strip() - generated_endpoint = str(openapi_result.path or "").strip() - diagnostics = dict(openapi_result.diagnostics or {}) - requested_fragment_type = self._requested_fragment_type(user_query, sub_intent) - has_operation_signal = bool(openapi_result.method) or bool(diagnostics.get("status_codes_found")) or bool(diagnostics.get("payload_description_found")) - has_contract_detail = any( - ( - openapi_result.request_schema is not None, - openapi_result.response_schema is not None, - diagnostics.get("status_codes_found"), - diagnostics.get("request_fields_found"), - diagnostics.get("response_fields_found"), - diagnostics.get("payload_description_found"), - ) - ) - if requested_endpoint and not exact_anchor_match: - return "degraded", "not_found_exact_anchor", False - if requested_endpoint and generated_endpoint and requested_endpoint != generated_endpoint: - return "degraded", "generated_endpoint_mismatch", False - if str(getattr(router_result.retrieval_spec.filters, "doc_type", "") or "") != "api_method": - return "degraded", "wrong_doc_type_match", False - if not generated_endpoint or not has_operation_signal: - return "degraded", "insufficient_docs_evidence", False - if sub_intent == "OPENAPI_FRAGMENT_GENERATE": - return self._evaluate_fragment_gate(requested_fragment_type, openapi_result) - if not openapi_result.method and not diagnostics.get("status_codes_found"): - return "degraded", "insufficient_docs_evidence", False - if not has_contract_detail: - return "degraded", "insufficient_docs_evidence", False - if openapi_result.request_schema is None or openapi_result.response_schema is None: - return "ready_partial", "answered_with_gaps", True - return "ready", "", True - - def _build_gate_details( - self, - *, - intent: str, - sub_intent: str, - answer_mode: str, - degraded_reason: str, - raw_rows: list[dict], - exact_anchor_match: bool, - matched_anchor_type: str, - matched_anchor_value: str | None, - openapi_result, - router_result, - ) -> tuple[str, str, list[str], list[str]]: - satisfied: list[str] = [] - missing: list[str] = [] - relation_hits = self._relation_hits(raw_rows) - if raw_rows: - satisfied.append("retrieval_non_empty") - else: - missing.append("retrieval_non_empty") - if matched_anchor_type in {"endpoint", "entity"} and matched_anchor_value: - if exact_anchor_match or (sub_intent == "RELATED_DOCS_EXPLAIN" and relation_hits > 0): - satisfied.append("exact_anchor_match") - else: - missing.append("exact_anchor_match") - if intent in {"OPENAPI_GENERATION", "OPENAPI_FROM_DOCUMENTATION"} and openapi_result is not None: - if openapi_result.path: - satisfied.append("path_found") - else: - missing.append("path_found") - diagnostics = openapi_result.diagnostics - if openapi_result.method: - satisfied.append("http_method_found") - else: - missing.append("http_method_found") - if diagnostics.get("operation_semantics_found"): - satisfied.append("operation_semantics_found") - else: - missing.append("operation_semantics_found") - if diagnostics.get("request_payload_found"): - satisfied.append("request_payload_found") - if diagnostics.get("response_payload_found"): - satisfied.append("response_payload_found") - if openapi_result.request_schema is not None: - satisfied.append("request_fields_found") - if openapi_result.response_schema is not None: - satisfied.append("response_fields_found") - if openapi_result.request_schema is None and openapi_result.response_schema is None: - missing.append("contract_fields_found") - if diagnostics.get("status_codes_found"): - satisfied.append("status_codes_found") - else: - missing.append("status_codes_found") - if diagnostics.get("payload_description_found"): - satisfied.append("payload_description_found") - else: - missing.append("payload_description_found") - if diagnostics.get("content_type_found"): - satisfied.append("content_type_found") - if diagnostics.get("examples_found"): - satisfied.append("examples_found") - if str(getattr(router_result.retrieval_spec.filters, "doc_type", "") or "") == "api_method": - satisfied.append("api_method_filter") - else: - missing.append("api_method_filter") - gate = answer_mode - if gate in {"diagnostic_only", "ready", "answered", "structured_spec"}: - gate = "allow" if not degraded_reason else "reject" - elif gate in {"ready_partial", "structured_spec_partial"}: - gate = "partial" - elif gate == "degraded": - gate = "reject" - return gate, self._gate_reason(gate, degraded_reason, sub_intent, relation_hits), missing, satisfied - - def _non_empty_layers(self, rows: list[dict], report: dict[str, Any], sub_intent: str) -> list[str]: - diagnostics = dict(report.get("layer_diagnostics") or {}) - if sub_intent == "RELATED_DOCS_EXPLAIN" and int(dict(diagnostics.get("D5_RELATION_GRAPH") or {}).get("hits") or 0) > 0: - return ["D5_RELATION_GRAPH"] - if diagnostics: - return [layer for layer, payload in diagnostics.items() if int(dict(payload).get("hits") or 0) > 0] - return self._layers_with_hits(rows) - - def _layers_with_hits(self, rows: list[dict]) -> list[str]: - result: list[str] = [] - for row in rows: - layer = str(row.get("layer") or "") - if layer.startswith("D") and layer not in result: - result.append(layer) - return result - - def _should_reject_on_exact_anchor( - self, - *, - sub_intent: str, - raw_rows: list[dict], - matched_anchor_type: str, - matched_anchor_value: str | None, - exact_anchor_match: bool, - ) -> bool: - if matched_anchor_type not in {"endpoint", "entity"} or not matched_anchor_value or exact_anchor_match: - return False - if sub_intent == "RELATED_DOCS_EXPLAIN" and self._has_relation_hits(raw_rows): - return False - return True - - def _has_relation_hits(self, rows: list[dict]) -> bool: - return self._relation_hits(rows) > 0 - - def _relation_hits(self, rows: list[dict]) -> int: - return sum(1 for row in rows if str(row.get("layer") or "") == "D5_RELATION_GRAPH") - - def _gate_reason(self, gate: str, degraded_reason: str, sub_intent: str, relation_hits: int) -> str: - if degraded_reason: - return degraded_reason - if sub_intent == "RELATED_DOCS_EXPLAIN" and relation_hits > 0: - return "relation_evidence_available" - if gate == "partial": - return "partial_evidence_available" - if gate == "allow": - return "evidence_sufficient" - return "" - - def _evaluate_fragment_gate(self, requested_fragment_type: str | None, openapi_result) -> tuple[str, str, bool]: - diagnostics = dict(openapi_result.diagnostics or {}) - if requested_fragment_type == "request_schema": - if openapi_result.request_schema is not None: - return "ready", "", True - if diagnostics.get("request_fields_found") or diagnostics.get("request_payload_found") or diagnostics.get("payload_description_found"): - return "ready_partial", "fragment_payload_only", True - return "degraded", "insufficient_docs_evidence", False - if requested_fragment_type == "response_schema": - if openapi_result.response_schema is not None: - return "ready", "", True - if diagnostics.get("response_fields_found") or diagnostics.get("response_payload_found") or diagnostics.get("status_codes_found"): - return "ready_partial", "fragment_response_partial", True - return "degraded", "insufficient_docs_evidence", False - if requested_fragment_type == "parameters": - if diagnostics.get("has_method") or diagnostics.get("content_type_found"): - return "ready_partial", "fragment_parameters_partial", True - return "degraded", "insufficient_docs_evidence", False - if openapi_result.request_schema is not None or openapi_result.response_schema is not None: - return "ready_partial", "answered_with_gaps", True - if diagnostics.get("payload_description_found"): - return "ready_partial", "fragment_payload_only", True - return "degraded", "insufficient_docs_evidence", False - - def _requested_fragment_type(self, user_query: str, sub_intent: str) -> str | None: - if sub_intent != "OPENAPI_FRAGMENT_GENERATE": - return None - lowered = user_query.lower() - if "request" in lowered: - return "request_schema" - if "response" in lowered: - return "response_schema" - if "parameter" in lowered or "парамет" in lowered: - return "parameters" - return "schema_fragment" - - def _fragment_evidence_found(self, requested_fragment_type: str | None, openapi_result) -> list[str]: - if openapi_result is None or requested_fragment_type is None: - return [] - diagnostics = dict(openapi_result.diagnostics or {}) - found: list[str] = [] - if openapi_result.path: - found.append("path") - if openapi_result.method: - found.append("method") - if diagnostics.get("operation_semantics_found"): - found.append("operation_semantics") - if diagnostics.get("payload_description_found"): - found.append("payload_description") - if diagnostics.get("status_codes_found"): - found.append("status_codes") - if requested_fragment_type == "request_schema" and openapi_result.request_schema is not None: - found.append("request_schema") - if requested_fragment_type == "request_schema" and diagnostics.get("request_fields_found"): - found.append("request_fields") - if requested_fragment_type == "response_schema" and openapi_result.response_schema is not None: - found.append("response_schema") - if requested_fragment_type == "response_schema" and diagnostics.get("response_fields_found"): - found.append("response_fields") - return found - - def _fragment_missing_requirements(self, requested_fragment_type: str | None, openapi_result) -> list[str]: - if openapi_result is None or requested_fragment_type is None: - return [] - diagnostics = dict(openapi_result.diagnostics or {}) - missing: list[str] = [] - if not openapi_result.path: - missing.append("path") - if requested_fragment_type == "request_schema": - if openapi_result.request_schema is None and not diagnostics.get("payload_description_found"): - missing.append("request_payload_evidence") - elif requested_fragment_type == "response_schema": - if openapi_result.response_schema is None and not diagnostics.get("status_codes_found"): - missing.append("response_payload_evidence") - elif openapi_result.request_schema is None and openapi_result.response_schema is None: - missing.append("schema_fragment") - return missing - - -def _default_conversation_state() -> Any: - from app.modules.agent.intent_router_v2 import ConversationState - - return ConversationState() - - -def _default_repo_context() -> Any: - from app.modules.agent.intent_router_v2 import RepoContext - - return RepoContext() - - -def _ms(started: float) -> int: - return int((perf_counter() - started) * 1000) diff --git a/src/app/modules/agent/runtime/docs_qa_pipeline/prompt_payload_builder.py b/src/app/modules/agent/runtime/docs_qa_pipeline/prompt_payload_builder.py deleted file mode 100644 index 2b85fda..0000000 --- a/src/app/modules/agent/runtime/docs_qa_pipeline/prompt_payload_builder.py +++ /dev/null @@ -1,35 +0,0 @@ -from __future__ import annotations - -import json - -from app.modules.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle, OpenAPIResult - - -class DocsPromptPayloadBuilder: - def build( - self, - *, - question: str, - intent: str, - sub_intent: str, - evidence_bundle: DocsEvidenceBundle, - api_contract: OpenAPIResult | None = None, - ) -> str: - payload = { - "question": question, - "documents": list(evidence_bundle.documents), - "facts": list(evidence_bundle.facts), - "entities": list(evidence_bundle.entities), - "workflows": list(evidence_bundle.workflows), - "relations": list(evidence_bundle.relations), - "chunks": list(evidence_bundle.chunks), - } - if api_contract is not None: - payload["api_contract"] = { - "path": api_contract.path, - "method": api_contract.method, - "request_schema": api_contract.request_schema, - "response_schema": api_contract.response_schema, - "diagnostics": dict(api_contract.diagnostics), - } - return json.dumps(payload, ensure_ascii=False, indent=2) diff --git a/src/app/modules/agent/runtime/executor.py b/src/app/modules/agent/runtime/executor.py deleted file mode 100644 index f1c655f..0000000 --- a/src/app/modules/agent/runtime/executor.py +++ /dev/null @@ -1,492 +0,0 @@ -"""Главный оркестратор runtime: роутинг → retrieval → evidence gate → генерация ответа (LLM).""" - -from __future__ import annotations - -import logging -from difflib import SequenceMatcher -from time import perf_counter - -from app.modules.agent.runtime.models import RuntimeDraftAnswer, RuntimeExecutionState, RuntimeFinalResult -from app.modules.agent.intent_router_v2 import ConversationState, IntentRouterV2 -from app.modules.agent.intent_router_v2.models import SymbolResolution -from app.modules.agent.runtime.steps.retrieval import RuntimeRetrievalAdapter, RuntimeRepoContextFactory -from app.modules.agent.runtime.steps.context import ( - build_answer_synthesis_input, - build_evidence_bundle, - build_retrieval_request, - build_retrieval_result, -) -from app.modules.agent.runtime.steps.gates.pre.evidence_gate import evaluate_evidence -from app.modules.agent.runtime.steps.gates.post.post_gate import RuntimePostEvidenceGate -from app.modules.agent.runtime.steps.answer_policy import RuntimeAnswerPolicy -from app.modules.agent.runtime.steps.generation import RuntimePromptPayloadBuilder, RuntimePromptSelector, RuntimeAnswerGenerator -from app.modules.agent.runtime.steps.finalization import RuntimeAnswerRepairService, assemble_final_result -from app.modules.agent.llm import AgentLlmService - -LOGGER = logging.getLogger(__name__) - - -class AgentRuntimeExecutor: - def __init__( - self, - llm: AgentLlmService | None, - *, - router: IntentRouterV2 | None = None, - retrieval: RuntimeRetrievalAdapter | None = None, - repo_context_factory: RuntimeRepoContextFactory | None = None, - prompt_selector: RuntimePromptSelector | None = None, - payload_builder: RuntimePromptPayloadBuilder | None = None, - answer_policy: RuntimeAnswerPolicy | None = None, - post_gate: RuntimePostEvidenceGate | None = None, - ) -> None: - self._llm = llm - self._router = router or IntentRouterV2() - self._retrieval = retrieval or RuntimeRetrievalAdapter() - self._repo_context_factory = repo_context_factory or RuntimeRepoContextFactory() - self._prompt_selector = prompt_selector or RuntimePromptSelector() - self._payload_builder = payload_builder or RuntimePromptPayloadBuilder() - self._answer_policy = answer_policy or RuntimeAnswerPolicy() - self._post_gate = post_gate or RuntimePostEvidenceGate() - self._repair = RuntimeAnswerRepairService(llm) if llm is not None else None - self._generator = RuntimeAnswerGenerator(llm) if llm is not None else None - - def execute(self, *, user_query: str, rag_session_id: str, files_map: dict[str, dict] | None = None) -> RuntimeFinalResult: - timings_ms: dict[str, int] = {} - runtime_trace: list[dict] = [] - answer_policy_branch = "" - decision_reason = "" - post_gate_snapshot: dict = {} - state = RuntimeExecutionState( - user_query=user_query, - rag_session_id=rag_session_id, - conversation_state=ConversationState(), - repo_context=self._repo_context_factory.build(files_map), - ) - started = perf_counter() - state.router_result = self._router.route(user_query, state.conversation_state, state.repo_context) - timings_ms["router"] = self._elapsed_ms(started) - runtime_trace.append( - { - "step": "router", - "status": "completed", - "timings_ms": {"router": timings_ms["router"]}, - "output": { - "intent": state.router_result.intent, - "sub_intent": state.router_result.query_plan.sub_intent, - "graph_id": state.router_result.graph_id, - "conversation_mode": state.router_result.conversation_mode, - }, - } - ) - state.retrieval_request = build_retrieval_request(state.router_result, rag_session_id) - started = perf_counter() - raw_rows = self._retrieve(state) - timings_ms["retrieval"] = self._elapsed_ms(started) - retrieval_report = self._retrieval.consume_retrieval_report() or {} - raw_rows, retrieval_report = self._hydrate_entrypoint_sources(state, raw_rows, retrieval_report) - symbol_resolution = self._resolve_symbol(state.router_result.symbol_resolution.model_dump(), raw_rows) - state.router_result = state.router_result.model_copy(update={"symbol_resolution": SymbolResolution(**symbol_resolution)}) - state.retrieval_result = build_retrieval_result(raw_rows, retrieval_report, symbol_resolution) - if state.retrieval_request.sub_intent.upper() == "EXPLAIN" and symbol_resolution.get("status") in {"not_found", "ambiguous"}: - state.retrieval_result = build_retrieval_result([], retrieval_report, symbol_resolution) - runtime_trace.append( - { - "step": "retrieval", - "status": "completed", - "timings_ms": {"retrieval": timings_ms["retrieval"]}, - "output": { - "rag_count": len(raw_rows), - "answer_path_rag_count": len(state.retrieval_result.raw_rows), - "resolved_symbol_status": symbol_resolution.get("status"), - "resolved_symbol": symbol_resolution.get("resolved_symbol"), - "requested_layers": list(state.retrieval_request.requested_layers or []), - }, - "diagnostics": retrieval_report or {}, - } - ) - state.evidence_pack = build_evidence_bundle(state.retrieval_result, state.router_result) - if state.retrieval_request.sub_intent.upper() == "EXPLAIN" and symbol_resolution.get("status") in {"not_found", "ambiguous"}: - state.evidence_pack.sufficient = False - state.evidence_pack.failure_reasons = ["target_not_resolved"] - pre_gate_input = self._build_pre_gate_input(state) - started = perf_counter() - gate_decision = evaluate_evidence(state.evidence_pack) - timings_ms["pre_evidence_gate"] = self._elapsed_ms(started) - state.answer_mode = "normal" if gate_decision.passed else "degraded" - state.degraded_message = gate_decision.degraded_message - runtime_trace.append( - { - "step": "pre_evidence_gate", - "status": "passed" if gate_decision.passed else "blocked", - "timings_ms": {"pre_evidence_gate": timings_ms["pre_evidence_gate"]}, - "input": pre_gate_input, - "output": { - "passed": gate_decision.passed, - "failure_reasons": list(gate_decision.failure_reasons), - "degraded_message": gate_decision.degraded_message, - "evidence_count": state.evidence_pack.evidence_count, - }, - } - ) - decision = self._answer_policy.decide(router_result=state.router_result, gate_decision=gate_decision) - answer_policy_branch = decision.branch - decision_reason = decision.reason - if not decision.should_call_llm: - state.answer_mode = decision.answer_mode - started = perf_counter() - runtime_trace.append( - { - "step": "llm", - "status": "skipped", - "timings_ms": {"llm": self._elapsed_ms(started)}, - "output": { - "reason": "policy_short_circuit", - "answer_mode": decision.answer_mode, - "decision_reason": decision.reason, - "answer_policy_branch": decision.branch, - }, - } - ) - started = perf_counter() - timings_ms["post_evidence_gate"] = self._elapsed_ms(started) - post_gate_snapshot = { - "input": { - "answer_mode": decision.answer_mode, - "draft_present": False, - "resolved_target": self._resolved_target(state), - }, - "output": {"reason": "no_draft_answer"}, - } - runtime_trace.append( - { - "step": "post_evidence_gate", - "status": "skipped", - "timings_ms": {"post_evidence_gate": timings_ms["post_evidence_gate"]}, - "input": post_gate_snapshot["input"], - "output": post_gate_snapshot["output"], - } - ) - return assemble_final_result( - state, - draft=None, - final_answer=decision.answer, - repair_used=False, - llm_used=False, - timings_ms=timings_ms, - runtime_trace=runtime_trace, - answer_policy_branch=answer_policy_branch, - decision_reason=decision_reason, - pre_gate_input=pre_gate_input, - post_gate_snapshot=post_gate_snapshot, - resolved_target=self._resolved_target(state), - post_gate=self._post_gate, - ) - if self._llm is None: - answer_policy_branch = "llm_unavailable" - decision_reason = "llm_service_missing" - started = perf_counter() - runtime_trace.append( - { - "step": "llm", - "status": "skipped", - "timings_ms": {"llm": self._elapsed_ms(started)}, - "output": {"reason": "llm_unavailable", "answer_policy_branch": answer_policy_branch, "decision_reason": decision_reason}, - } - ) - started = perf_counter() - timings_ms["post_evidence_gate"] = self._elapsed_ms(started) - post_gate_snapshot = { - "input": { - "answer_mode": state.answer_mode, - "draft_present": False, - "resolved_target": self._resolved_target(state), - }, - "output": {"reason": "no_draft_answer"}, - } - runtime_trace.append( - { - "step": "post_evidence_gate", - "status": "skipped", - "timings_ms": {"post_evidence_gate": timings_ms["post_evidence_gate"]}, - "input": post_gate_snapshot["input"], - "output": post_gate_snapshot["output"], - } - ) - return assemble_final_result( - state, - draft=None, - final_answer="", - repair_used=False, - llm_used=False, - timings_ms=timings_ms, - runtime_trace=runtime_trace, - answer_policy_branch=answer_policy_branch, - decision_reason=decision_reason, - pre_gate_input=pre_gate_input, - post_gate_snapshot=post_gate_snapshot, - resolved_target=self._resolved_target(state), - post_gate=self._post_gate, - ) - state.synthesis_input = build_answer_synthesis_input(user_query, state.evidence_pack) - prompt_name = self._prompt_selector.select( - intent=state.router_result.intent, - sub_intent=state.retrieval_request.sub_intent, - answer_mode=state.answer_mode, - ) - prompt_payload = self._payload_builder.build( - user_query=user_query, - synthesis_input=state.synthesis_input, - evidence_pack=state.evidence_pack, - answer_mode=state.answer_mode, - ) - started = perf_counter() - draft = RuntimeDraftAnswer( - prompt_name=prompt_name, - prompt_payload=prompt_payload, - answer=self._generator.generate(prompt_name, prompt_payload), - ) - timings_ms["llm"] = self._elapsed_ms(started) - runtime_trace.append( - { - "step": "llm", - "status": "completed", - "timings_ms": {"llm": timings_ms["llm"]}, - "output": { - "prompt_name": prompt_name, - "answer_preview": draft.answer[:300], - "resolved_target": self._resolved_target(state), - "answer_policy_branch": answer_policy_branch, - "decision_reason": decision_reason, - }, - } - ) - post_gate_input = { - "answer_mode": state.answer_mode, - "degraded_message": state.degraded_message, - "resolved_target": self._resolved_target(state), - "draft_answer_preview": draft.answer[:300], - "repair_candidate": bool(self._repair is not None), - } - started = perf_counter() - validation = self._post_gate.validate( - answer=draft.answer, - answer_mode=state.answer_mode, - degraded_message=state.degraded_message, - sub_intent=state.retrieval_request.sub_intent, - user_query=user_query, - evidence_pack=state.evidence_pack, - ) - timings_ms["post_evidence_gate"] = self._elapsed_ms(started) - final_answer = draft.answer - repair_used = False - if not validation.passed and self._repair is not None: - started = perf_counter() - final_answer = self._repair.repair(draft_answer=draft.answer, validation=validation, prompt_payload=prompt_payload) - repair_used = True - timings_ms["repair"] = self._elapsed_ms(started) - started = perf_counter() - validation = self._post_gate.validate( - answer=final_answer, - answer_mode=state.answer_mode, - degraded_message=state.degraded_message, - sub_intent=state.retrieval_request.sub_intent, - user_query=user_query, - evidence_pack=state.evidence_pack, - ) - timings_ms["post_evidence_gate_recheck"] = self._elapsed_ms(started) - if not validation.passed: - final_answer = self._fallback_answer(state) - state.answer_mode = self._fallback_mode(state) - post_gate_snapshot = { - "input": post_gate_input, - "output": { - "passed": validation.passed, - "action": validation.action, - "reasons": list(validation.reasons), - "repair_used": repair_used, - "final_answer_preview": final_answer[:300], - }, - } - runtime_trace.append( - { - "step": "post_evidence_gate", - "status": "passed" if validation.passed else "failed", - "timings_ms": { - "post_evidence_gate": timings_ms["post_evidence_gate"], - "post_evidence_gate_recheck": timings_ms.get("post_evidence_gate_recheck", 0), - "repair": timings_ms.get("repair", 0), - }, - "input": post_gate_snapshot["input"], - "output": post_gate_snapshot["output"], - } - ) - return assemble_final_result( - state, - draft=draft, - final_answer=final_answer, - repair_used=repair_used, - llm_used=True, - validation=validation, - timings_ms=timings_ms, - runtime_trace=runtime_trace, - answer_policy_branch=answer_policy_branch, - decision_reason=decision_reason, - pre_gate_input=pre_gate_input, - post_gate_snapshot=post_gate_snapshot, - resolved_target=self._resolved_target(state), - post_gate=self._post_gate, - ) - - def _retrieve(self, state: RuntimeExecutionState) -> list[dict]: - assert state.retrieval_request is not None - if state.retrieval_request.sub_intent == "OPEN_FILE" and state.retrieval_request.path_scope: - return self._retrieval.retrieve_exact_files( - state.rag_session_id, - paths=state.retrieval_request.path_scope, - layers=["C0_SOURCE_CHUNKS"], - limit=200, - query=state.retrieval_request.query, - ranking_profile=str(getattr(state.retrieval_request.retrieval_spec, "rerank_profile", "") or ""), - ) - return self._retrieval.retrieve_with_plan( - state.rag_session_id, - state.retrieval_request.query, - state.retrieval_request.retrieval_spec, - state.retrieval_request.retrieval_constraints, - query_plan=state.retrieval_request.query_plan, - ) - - def _resolve_symbol(self, initial: dict, rag_rows: list[dict]) -> dict: - if str(initial.get("status") or "") != "pending": - return initial - candidates = [str(item).strip() for item in initial.get("alternatives", []) if str(item).strip()] - found = [ - str(row.get("title") or "").strip() - for row in rag_rows - if str(row.get("layer") or "") == "C1_SYMBOL_CATALOG" and str(row.get("title") or "").strip() - ] - exact = next((item for item in found if item in candidates), None) - if exact: - return {"status": "resolved", "resolved_symbol": exact, "alternatives": found[:5], "confidence": 0.99} - close = self._close_matches(candidates, found) - if close: - return {"status": "ambiguous", "resolved_symbol": None, "alternatives": close[:5], "confidence": 0.55} - return {"status": "not_found", "resolved_symbol": None, "alternatives": close[:5], "confidence": 0.0} - - def _elapsed_ms(self, started: float) -> int: - return max(1, round((perf_counter() - started) * 1000)) - - def _build_pre_gate_input(self, state: RuntimeExecutionState) -> dict: - evidence = state.evidence_pack - retrieval = state.retrieval_result - return { - "resolved_target": self._resolved_target(state), - "sub_intent": state.retrieval_request.sub_intent if state.retrieval_request else None, - "target_type": evidence.target_type if evidence else None, - "evidence_count": evidence.evidence_count if evidence else 0, - "code_chunk_count": len(evidence.code_chunks) if evidence else 0, - "entrypoint_count": len(evidence.entrypoints) if evidence else 0, - "relation_count": len(evidence.relations) if evidence else 0, - "test_evidence_count": len(evidence.test_evidence) if evidence else 0, - "symbol_resolution_status": retrieval.symbol_resolution_status if retrieval else None, - "path_scope": list(state.retrieval_request.path_scope) if state.retrieval_request else [], - } - - def _resolved_target(self, state: RuntimeExecutionState) -> str | None: - if state.evidence_pack and state.evidence_pack.resolved_target: - return state.evidence_pack.resolved_target - if state.retrieval_result and state.retrieval_result.resolved_symbol: - return state.retrieval_result.resolved_symbol - if state.retrieval_request and state.retrieval_request.path_scope: - return state.retrieval_request.path_scope[0] - return None - - def _close_matches(self, candidates: list[str], found: list[str]) -> list[str]: - ranked: list[tuple[float, str]] = [] - for candidate in candidates: - for item in found: - score = SequenceMatcher(None, candidate.lower(), item.lower()).ratio() - if score >= 0.52: - ranked.append((score, item)) - ranked.sort(key=lambda pair: (-pair[0], pair[1])) - result: list[str] = [] - for _, item in ranked: - if item not in result: - result.append(item) - return result - - def _hydrate_entrypoint_sources( - self, - state: RuntimeExecutionState, - raw_rows: list[dict], - retrieval_report: dict, - ) -> tuple[list[dict], dict]: - if not state.retrieval_request or state.retrieval_request.sub_intent.upper() != "FIND_ENTRYPOINTS": - return raw_rows, retrieval_report - entrypoint_paths = [] - for row in raw_rows: - if str(row.get("layer") or "") != "C3_ENTRYPOINTS": - continue - path = str(row.get("path") or "").strip() - if path and path not in entrypoint_paths: - entrypoint_paths.append(path) - if not entrypoint_paths: - return raw_rows, retrieval_report - extra_rows = self._retrieval.retrieve_exact_files( - state.rag_session_id, - paths=entrypoint_paths[:3], - layers=["C0_SOURCE_CHUNKS"], - limit=24, - query=state.user_query, - ranking_profile="entrypoint_source_hydration", - ) - extra_report = self._retrieval.consume_retrieval_report() or {} - return self._merge_rows(raw_rows, extra_rows), self._merge_reports(retrieval_report, extra_report) - - def _merge_rows(self, base_rows: list[dict], extra_rows: list[dict]) -> list[dict]: - merged: list[dict] = [] - seen: set[tuple[str, str, str, int | None, int | None]] = set() - for row in [*base_rows, *extra_rows]: - key = ( - str(row.get("layer") or ""), - str(row.get("path") or ""), - str(row.get("title") or ""), - row.get("span_start"), - row.get("span_end"), - ) - if key in seen: - continue - seen.add(key) - merged.append(row) - return merged - - def _merge_reports(self, base: dict, extra: dict) -> dict: - merged = dict(base or {}) - merged["executed_layers"] = list(dict.fromkeys([*(base.get("executed_layers") or []), *(extra.get("executed_layers") or [])])) - merged["retrieval_mode_by_layer"] = {**dict(base.get("retrieval_mode_by_layer") or {}), **dict(extra.get("retrieval_mode_by_layer") or {})} - merged["top_k_by_layer"] = {**dict(base.get("top_k_by_layer") or {}), **dict(extra.get("top_k_by_layer") or {})} - merged["filters_by_layer"] = {**dict(base.get("filters_by_layer") or {}), **dict(extra.get("filters_by_layer") or {})} - merged["retrieval_by_layer_ms"] = {**dict(base.get("retrieval_by_layer_ms") or {}), **dict(extra.get("retrieval_by_layer_ms") or {})} - merged["fallback"] = dict(extra.get("fallback") or base.get("fallback") or {"used": False, "reason": None}) - merged["supplemental_requests"] = [*(base.get("supplemental_requests") or []), *(extra.get("requests") or [])] - return merged - - def _fallback_mode(self, state: RuntimeExecutionState) -> str: - status = str(state.router_result.symbol_resolution.status if state.router_result and state.router_result.symbol_resolution else "") - if status == "ambiguous": - return "ambiguous" - if status == "not_found": - return "not_found" - return "degraded" - - def _fallback_answer(self, state: RuntimeExecutionState) -> str: - symbol_resolution = state.router_result.symbol_resolution if state.router_result else None - query_plan = state.router_result.query_plan if state.router_result else None - target = next((item for item in list(query_plan.symbol_candidates or []) if item), "запрошенная сущность") if query_plan else "запрошенная сущность" - if symbol_resolution and symbol_resolution.status == "ambiguous": - return self._answer_policy.decide(router_result=state.router_result, gate_decision=evaluate_evidence(state.evidence_pack)).answer - if symbol_resolution and symbol_resolution.status == "not_found": - return self._answer_policy.decide(router_result=state.router_result, gate_decision=evaluate_evidence(state.evidence_pack)).answer - if state.degraded_message: - return state.degraded_message - return f"Недостаточно подтверждённых данных для уверенного ответа по {target}." diff --git a/src/app/modules/agent/runtime/legacy_pipeline.py b/src/app/modules/agent/runtime/legacy_pipeline.py deleted file mode 100644 index 1eb4d72..0000000 --- a/src/app/modules/agent/runtime/legacy_pipeline.py +++ /dev/null @@ -1,232 +0,0 @@ -"""Legacy test-first CODE_QA pipeline: router → retrieval → evidence → synthesis → diagnostics. Prefer agent.runtime.executor.""" - -from __future__ import annotations - -from dataclasses import dataclass, field -from difflib import get_close_matches -from time import perf_counter -from typing import Any, Protocol - -from app.modules.agent.runtime.steps.context import ( - build_answer_synthesis_input, - build_diagnostics_report, - build_evidence_bundle, - build_retrieval_request, - build_retrieval_result, - EvidenceBundle, - RetrievalRequest, - RetrievalResult, - RouterResult, -) -from app.modules.agent.runtime.steps.gates.pre.evidence_gate import evaluate_evidence - - -class RetrievalAdapter(Protocol): - def retrieve_with_plan( - self, - rag_session_id: str, - query: str, - retrieval_spec: Any, - retrieval_constraints: Any = None, - *, - query_plan: Any = None, - ) -> list[dict]: ... - - def retrieve_exact_files( - self, - rag_session_id: str, - *, - repo_id: str | None = None, - paths: list[str], - layers: list[str] | None = None, - limit: int = 200, - query: str = "", - ranking_profile: str = "", - ) -> list[dict]: ... - - def hydrate_resolved_symbol_sources( - self, - rag_session_id: str, - base_query: str, - rag_rows: list[dict], - symbol_resolution: dict, - retrieval_spec: Any, - retrieval_constraints: Any = None, - ) -> list[dict]: ... - - def force_symbol_context_c0( - self, - rag_session_id: str, - *, - rag_rows: list[dict], - symbol_resolution: dict, - limit: int = 20, - ) -> list[dict]: ... - - def consume_retrieval_report(self) -> dict[str, Any] | None: ... - - -@dataclass(slots=True) -class CodeQAPipelineResult: - user_query: str - rag_session_id: str - router_result: RouterResult - retrieval_request: RetrievalRequest - retrieval_result: RetrievalResult - evidence_bundle: EvidenceBundle - evidence_gate_passed: bool - answer_synthesis_input: Any - diagnostics_report: Any - answer_mode: str = "normal" - degraded_message: str = "" - llm_answer: str | None = None - timings_ms: dict[str, int] = field(default_factory=dict) - - -class CodeQAPipelineRunner: - """Legacy test-first CODE_QA pipeline entrypoint. Prefer AgentRuntimeExecutor.""" - - def __init__( - self, - router: Any, - retrieval_adapter: RetrievalAdapter, - repo_context: Any = None, - ) -> None: - self._router = router - self._adapter = retrieval_adapter - self._repo_context = repo_context - - def run( - self, - user_query: str, - rag_session_id: str, - *, - conversation_state: Any = None, - run_retrieval: bool = True, - run_hydrate: bool = True, - ) -> CodeQAPipelineResult: - timings: dict[str, int] = {} - t0 = perf_counter() - router_result = self._router.route( - user_query, - conversation_state or _default_conversation_state(), - self._repo_context or _default_repo_context(), - ) - timings["router"] = _ms(t0) - - request = build_retrieval_request(router_result, rag_session_id) - raw_rows: list[dict] = [] - symbol_resolution = router_result.symbol_resolution.model_dump() if router_result.symbol_resolution else {} - - if run_retrieval: - t1 = perf_counter() - raw_rows = self._retrieve(router_result, request) - timings["retrieval_total"] = _ms(t1) - symbol_resolution = self._resolve_symbol(symbol_resolution, raw_rows) - timings["symbol_resolution"] = 0 - if run_hydrate and request.sub_intent == "EXPLAIN" and symbol_resolution.get("status") == "resolved": - raw_rows = self._adapter.hydrate_resolved_symbol_sources( - rag_session_id, - request.query, - raw_rows, - symbol_resolution, - request.retrieval_spec, - request.retrieval_constraints, - ) - raw_rows = self._adapter.force_symbol_context_c0( - rag_session_id, - rag_rows=raw_rows, - symbol_resolution=symbol_resolution, - limit=20, - ) - - report = self._adapter.consume_retrieval_report() or {} - retrieval_result = build_retrieval_result(raw_rows, report, symbol_resolution) - evidence_bundle = build_evidence_bundle(retrieval_result, router_result) - gate_decision = evaluate_evidence(evidence_bundle) - synthesis_input = build_answer_synthesis_input(user_query, evidence_bundle) - answer_mode = "normal" if gate_decision.passed else "degraded" - if not gate_decision.passed and evidence_bundle.evidence_count == 0: - answer_mode = "insufficient" - diagnostics_report = build_diagnostics_report( - router_result=router_result, - retrieval_request=request, - retrieval_result=retrieval_result, - evidence_bundle=evidence_bundle, - answer_mode=answer_mode, - timings_ms=timings, - ) - - return CodeQAPipelineResult( - user_query=user_query, - rag_session_id=rag_session_id, - router_result=router_result, - retrieval_request=request, - retrieval_result=retrieval_result, - evidence_bundle=evidence_bundle, - evidence_gate_passed=gate_decision.passed, - answer_synthesis_input=synthesis_input, - diagnostics_report=diagnostics_report, - answer_mode=answer_mode, - degraded_message=gate_decision.degraded_message, - timings_ms=timings, - ) - - def _retrieve(self, router_result: RouterResult, request: RetrievalRequest) -> list[dict]: - path_scope = list(request.path_scope) - spec = request.retrieval_spec - if request.sub_intent == "OPEN_FILE" and path_scope: - return self._adapter.retrieve_exact_files( - request.rag_session_id, - paths=path_scope, - layers=["C0_SOURCE_CHUNKS"], - limit=200, - query=request.query, - ranking_profile=getattr(spec, "rerank_profile", "") or "", - ) - return self._adapter.retrieve_with_plan( - request.rag_session_id, - request.query, - spec, - request.retrieval_constraints, - query_plan=request.query_plan, - ) - - def _resolve_symbol(self, initial: dict, rag_rows: list[dict]) -> dict: - status = str(initial.get("status") or "not_requested") - if status != "pending": - return initial - candidates = [str(x).strip() for x in initial.get("alternatives", []) if str(x).strip()] - c1_titles = list(dict.fromkeys( - str(row.get("title") or "").strip() - for row in rag_rows - if str(row.get("layer") or "") == "C1_SYMBOL_CATALOG" and str(row.get("title") or "").strip() - )) - if not c1_titles: - return {**initial, "status": "not_found"} - exact = next((t for t in c1_titles if t in candidates), None) - if exact: - return {"status": "resolved", "resolved_symbol": exact, "alternatives": c1_titles[:5], "confidence": 0.99} - fuzzy = [] - for c in candidates: - fuzzy.extend(get_close_matches(c, c1_titles, n=3, cutoff=0.7)) - fuzzy = list(dict.fromkeys(fuzzy)) - if len(fuzzy) == 1: - return {"status": "resolved", "resolved_symbol": fuzzy[0], "alternatives": c1_titles[:5], "confidence": 0.82} - if len(fuzzy) > 1: - return {"status": "ambiguous", "resolved_symbol": None, "alternatives": fuzzy[:5], "confidence": 0.55} - return {"status": "not_found", "resolved_symbol": None, "alternatives": c1_titles[:5], "confidence": 0.0} - - -def _ms(started: float) -> int: - return int((perf_counter() - started) * 1000) - - -def _default_conversation_state() -> Any: - from app.modules.agent.intent_router_v2 import ConversationState - return ConversationState() - - -def _default_repo_context() -> Any: - from app.modules.agent.intent_router_v2 import RepoContext - return RepoContext() diff --git a/src/app/modules/agent/runtime/models.py b/src/app/modules/agent/runtime/models.py deleted file mode 100644 index daf2644..0000000 --- a/src/app/modules/agent/runtime/models.py +++ /dev/null @@ -1,60 +0,0 @@ -"""Модели состояния и результата runtime-оркестратора.""" - -from __future__ import annotations - -from typing import Any - -from pydantic import BaseModel, ConfigDict, Field - -from app.modules.agent.runtime.steps.context.contracts import ( - AnswerSynthesisInput, - DiagnosticsReport, - EvidenceBundle, - RetrievalRequest, - RetrievalResult, -) -from app.modules.agent.runtime.steps.gates.post.post_gate import RuntimeValidationResult -from app.modules.agent.intent_router_v2.models import ConversationState, IntentRouterResult, RepoContext - - -class RuntimeDraftAnswer(BaseModel): - model_config = ConfigDict(extra="forbid") - - prompt_name: str - prompt_payload: str - answer: str = "" - - -class RuntimeFinalResult(BaseModel): - model_config = ConfigDict(extra="forbid") - - final_answer: str - answer_mode: str = "normal" - repair_used: bool = False - llm_used: bool = False - draft_answer: RuntimeDraftAnswer | None = None - validation: RuntimeValidationResult = Field(default_factory=RuntimeValidationResult) - router_result: IntentRouterResult | None = None - retrieval_request: RetrievalRequest | None = None - retrieval_result: RetrievalResult | None = None - evidence_pack: EvidenceBundle | None = None - diagnostics: DiagnosticsReport - runtime_trace: list[dict[str, Any]] = Field(default_factory=list) - - -class RuntimeExecutionState(BaseModel): - model_config = ConfigDict(extra="forbid") - - user_query: str - rag_session_id: str - conversation_state: ConversationState = Field(default_factory=ConversationState) - repo_context: RepoContext = Field(default_factory=RepoContext) - router_result: IntentRouterResult | None = None - retrieval_request: RetrievalRequest | None = None - retrieval_result: RetrievalResult | None = None - evidence_pack: EvidenceBundle | None = None - synthesis_input: AnswerSynthesisInput | None = None - diagnostics: DiagnosticsReport | None = None - answer_mode: str = "normal" - degraded_message: str = "" - final_result: RuntimeFinalResult | None = None diff --git a/src/app/modules/agent/runtime/steps/answer_policy/__init__.py b/src/app/modules/agent/runtime/steps/answer_policy/__init__.py deleted file mode 100644 index 6e4b61a..0000000 --- a/src/app/modules/agent/runtime/steps/answer_policy/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -"""Политика ответа: вызов LLM или короткий ответ по результату evidence gate.""" - -from app.modules.agent.runtime.steps.answer_policy.policy import RuntimeAnswerPolicy, RuntimePolicyDecision -from app.modules.agent.runtime.steps.answer_policy.short_answer_formatter import RuntimeShortAnswerFormatter - -__all__ = ["RuntimeAnswerPolicy", "RuntimePolicyDecision", "RuntimeShortAnswerFormatter"] diff --git a/src/app/modules/agent/runtime/steps/answer_policy/policy.py b/src/app/modules/agent/runtime/steps/answer_policy/policy.py deleted file mode 100644 index c2c1931..0000000 --- a/src/app/modules/agent/runtime/steps/answer_policy/policy.py +++ /dev/null @@ -1,73 +0,0 @@ -"""Политика принятия решения: вызывать LLM или вернуть короткий ответ по evidence gate.""" - -from __future__ import annotations - -from dataclasses import dataclass - -from app.modules.agent.runtime.steps.gates.pre.evidence_gate import EvidenceGateDecision -from app.modules.agent.intent_router_v2.models import IntentRouterResult -from app.modules.agent.runtime.steps.answer_policy.short_answer_formatter import RuntimeShortAnswerFormatter - - -@dataclass(slots=True, frozen=True) -class RuntimePolicyDecision: - answer_mode: str - answer: str = "" - should_call_llm: bool = True - branch: str = "normal_answer" - reason: str = "evidence_sufficient" - - -class RuntimeAnswerPolicy: - def __init__(self, formatter: RuntimeShortAnswerFormatter | None = None) -> None: - self._formatter = formatter or RuntimeShortAnswerFormatter() - - def decide( - self, - *, - router_result: IntentRouterResult, - gate_decision: EvidenceGateDecision, - ) -> RuntimePolicyDecision: - sub_intent = router_result.query_plan.sub_intent.upper() - symbol_resolution = router_result.symbol_resolution - if sub_intent == "OPEN_FILE" and "path_scope_empty" in gate_decision.failure_reasons: - path_scope = list(getattr(router_result.retrieval_spec.filters, "path_scope", []) or []) - target = path_scope[0] if path_scope else "запрошенный файл" - return RuntimePolicyDecision( - answer_mode="not_found", - answer=self._formatter.open_file_not_found(target), - should_call_llm=False, - branch="open_file_not_found", - reason="path_scope_empty", - ) - if sub_intent == "EXPLAIN" and symbol_resolution.status == "not_found": - return RuntimePolicyDecision( - answer_mode="not_found", - answer=self._formatter.entity_not_found(self._target_label(router_result), symbol_resolution.alternatives), - should_call_llm=False, - branch="explain_not_found", - reason="symbol_resolution_not_found", - ) - if sub_intent == "EXPLAIN" and symbol_resolution.status == "ambiguous": - return RuntimePolicyDecision( - answer_mode="ambiguous", - answer=self._formatter.entity_ambiguous(self._target_label(router_result), symbol_resolution.alternatives), - should_call_llm=False, - branch="explain_ambiguous_symbol", - reason="symbol_resolution_ambiguous", - ) - if not gate_decision.passed: - answer_mode = "insufficient" if "insufficient_evidence" in gate_decision.failure_reasons else "degraded" - reason = gate_decision.failure_reasons[0] if gate_decision.failure_reasons else "evidence_gate_failed" - return RuntimePolicyDecision( - answer_mode=answer_mode, - answer=self._formatter.insufficient(gate_decision.degraded_message), - should_call_llm=False, - branch="evidence_gate_short_circuit", - reason=reason, - ) - return RuntimePolicyDecision(answer_mode="normal", branch="normal_answer", reason="evidence_sufficient") - - def _target_label(self, router_result: IntentRouterResult) -> str: - candidates = [item.strip() for item in list(router_result.query_plan.symbol_candidates or []) if item and item.strip()] - return candidates[0] if candidates else "запрошенная сущность" diff --git a/src/app/modules/agent/runtime/steps/answer_policy/short_answer_formatter.py b/src/app/modules/agent/runtime/steps/answer_policy/short_answer_formatter.py deleted file mode 100644 index 73233d0..0000000 --- a/src/app/modules/agent/runtime/steps/answer_policy/short_answer_formatter.py +++ /dev/null @@ -1,35 +0,0 @@ -"""Форматирование коротких ответов для режимов not_found, ambiguous, insufficient.""" - -from __future__ import annotations - - -class RuntimeShortAnswerFormatter: - def open_file_not_found(self, target: str) -> str: - return f"Файл {target} не найден." - - def entity_not_found(self, target: str, alternatives: list[str]) -> str: - base = f"Сущность {target} не найдена в доступном коде." - suffix = self._alternatives(alternatives) - return f"{base} {suffix}".strip() - - def entity_ambiguous(self, target: str, alternatives: list[str]) -> str: - base = f"Сущность {target} не удалось однозначно разрешить." - suffix = self._alternatives(alternatives) - return f"{base} {suffix}".strip() - - def insufficient(self, message: str) -> str: - normalized = (message or "").strip() - if normalized: - return normalized - return "Недостаточно подтверждённых данных для уверенного ответа." - - def related_only(self, target: str, alternatives: list[str]) -> str: - base = f"Прямых подтверждений для сущности {target} не найдено." - suffix = self._alternatives(alternatives) - return f"{base} {suffix}".strip() - - def _alternatives(self, alternatives: list[str]) -> str: - items = [item.strip() for item in alternatives if item and item.strip()] - if not items: - return "" - return f"Близкие варианты: {', '.join(items[:3])}." diff --git a/src/app/modules/agent/runtime/steps/context/__init__.py b/src/app/modules/agent/runtime/steps/context/__init__.py deleted file mode 100644 index c3fa187..0000000 --- a/src/app/modules/agent/runtime/steps/context/__init__.py +++ /dev/null @@ -1,35 +0,0 @@ -"""Контракты и билдеры контекста пайплайна: retrieval request/result, evidence bundle, diagnostics, synthesis.""" - -from app.modules.agent.runtime.steps.context.contracts import ( - AnswerSynthesisInput, - CodeChunkItem, - DiagnosticsReport, - EvidenceBundle, - FailureReason, - RetrievalRequest, - RetrievalResult, - RouterResult, -) -from app.modules.agent.runtime.steps.context.retrieval_request_builder import build_retrieval_request -from app.modules.agent.runtime.steps.context.retrieval_result_builder import build_retrieval_result -from app.modules.agent.runtime.steps.context.evidence_bundle_builder import build_evidence_bundle -from app.modules.agent.runtime.steps.context.diagnostics import build_diagnostics_report -from app.modules.agent.runtime.steps.context.answer_synthesis import build_answer_synthesis_input -from app.modules.agent.runtime.steps.context.answer_fact_curator import build_curated_answer_facts - -__all__ = [ - "AnswerSynthesisInput", - "CodeChunkItem", - "DiagnosticsReport", - "EvidenceBundle", - "FailureReason", - "RetrievalRequest", - "RetrievalResult", - "RouterResult", - "build_retrieval_request", - "build_retrieval_result", - "build_evidence_bundle", - "build_diagnostics_report", - "build_answer_synthesis_input", - "build_curated_answer_facts", -] diff --git a/src/app/modules/agent/runtime/steps/context/answer_fact_curator.py b/src/app/modules/agent/runtime/steps/context/answer_fact_curator.py deleted file mode 100644 index 75553ea..0000000 --- a/src/app/modules/agent/runtime/steps/context/answer_fact_curator.py +++ /dev/null @@ -1,345 +0,0 @@ -"""Курация фактов из EvidenceBundle для сценариев explain, architecture, trace_flow.""" - -from __future__ import annotations - -import re -from typing import Any - -from app.modules.agent.runtime.steps.context.contracts import CodeChunkItem, EvidenceBundle - -_CALL_RE = re.compile(r"([A-Za-z_][\w\.]*)\s*\(") -_FIELD_RE = re.compile(r"self\.(\w+)") -_SIGNATURE_RE = re.compile(r"(?P[A-Za-z_][\w\.]*)\((?P[^)]*)\)") -_RETRIEVAL_LABELS = ("dataflow_slice", "execution_trace", "trace_path") -_RELATION_VERBS = { - "calls": "вызывает", - "instantiates": "создает", - "inherits": "наследует", - "imports": "импортирует", - "reads_attr": "читает", - "writes_attr": "записывает", -} - - -def build_curated_answer_facts(bundle: EvidenceBundle) -> dict[str, Any]: - target = str(bundle.resolved_target or "").strip() - semantic_hints = _semantic_hints(bundle.code_chunks) - primary_chunks = [chunk for chunk in bundle.code_chunks if chunk.layer != "C4_SEMANTIC_ROLES"] - relations = _normalized_relations(bundle.relations) - target_relations = [relation for relation in relations if _is_target_relation(relation, target)] - - return { - "scenario": (bundle.resolved_sub_intent or "EXPLAIN").upper(), - "semantic_hints": semantic_hints, - "primary_chunk_count": len(primary_chunks), - "relation_count": len(relations), - "explain": _explain_facts(bundle, primary_chunks, target_relations), - "architecture": _architecture_facts(bundle, primary_chunks, target_relations), - "trace_flow": _trace_flow_facts(bundle, primary_chunks, target_relations), - } - - -def _explain_facts(bundle: EvidenceBundle, chunks: list[CodeChunkItem], relations: list[dict[str, Any]]) -> dict[str, Any]: - target = str(bundle.resolved_target or "").strip() - signatures = [_signature_payload(chunk) for chunk in chunks if chunk.layer == "C1_SYMBOL_CATALOG"] - target_signatures = [item for item in signatures if _is_target_symbol(item["name"], target)] - # Конкретные имена методов: полный qname и короткое отображаемое имя (Class.method или method) - methods_full = _unique(item["name"] for item in target_signatures if item["kind"] == "method") - methods_short = _unique(_short_method_name(name) for name in methods_full) - methods = methods_short or methods_full[:6] - constructor_args = _unique( - arg - for item in target_signatures - if item["name"].endswith(".__init__") - for arg in item["args"] - if arg not in {"self", "cls"} - ) - # Конкретные вызовы: из relations и fallback из кода - calls = _unique( - _display_call_target(relation["target"]) - for relation in relations - if relation["edge_type"] in {"calls", "instantiates"} - ) - if not calls: - calls = _fallback_calls(chunks, target) - # Поля: из relations и из self.attr в коде - fields = _unique( - relation["target"].split(".", 1)[-1] - for relation in relations - if relation["edge_type"] in {"reads_attr", "writes_attr"} - ) - if not fields: - fields = _unique(field for chunk in chunks if _chunk_matches_target(chunk, target) for field in _FIELD_RE.findall(chunk.content or "")) - # Зависимости: импорты и инстанциации - dependencies = _unique( - _display_dependency_target(relation["target"]) - for relation in relations - if relation["edge_type"] in {"imports", "instantiates"} - ) - required_files = _unique(chunk.path for chunk in chunks if _chunk_matches_target(chunk, target) and chunk.path) - required_symbols = _unique(item["name"] for item in target_signatures if item["name"]) or _unique([target] if target else bundle.target_symbol_candidates) - - fact_gaps: list[str] = [] - if not methods: - fact_gaps.append("Методы целевой сущности не подтверждены в извлеченных C0/C1 данных.") - if not calls: - fact_gaps.append("Конкретные вызовы целевой сущности не подтверждены в C2/C0.") - if not dependencies: - fact_gaps.append("Явные зависимости целевой сущности не подтверждены.") - if methods and not (calls or dependencies): - fact_gaps.append("Есть методы, но вызовы/зависимости не извлечены — опирайся только на перечисленные методы и поля.") - - return { - "required_symbols": required_symbols[:8], - "required_methods": methods[:8], - "required_calls": calls[:8], - "required_fields": fields[:8], - "required_constructor_args": constructor_args[:8], - "required_dependencies": dependencies[:8], - "required_files": required_files[:4], - "fact_gaps": fact_gaps, - } - - -def _architecture_facts(bundle: EvidenceBundle, chunks: list[CodeChunkItem], relations: list[dict[str, Any]]) -> dict[str, Any]: - target = str(bundle.resolved_target or "").strip() - # Компоненты: target, из связей, из заголовков чанков (классы/модули) - from_relations = _unique( - [_component_name(relation["source"]) for relation in relations] - + [_component_name(relation["target"]) for relation in relations] - ) - from_chunks = _unique( - _component_name(chunk.title) or _component_name(str(dict(chunk.metadata or {}).get("qname") or "")) - for chunk in chunks - if chunk.layer in ("C1_SYMBOL_CATALOG", "C0_SOURCE_CHUNKS") and (target and _chunk_matches_target(chunk, target)) - ) - components = _unique(([target] if target else []) + from_relations + from_chunks) - relation_rows = [ - { - "source": _component_name(relation["source"]), - "verb": relation["verb"], - "target": _component_name(relation["target"]), - "source_method": relation["source"], - "target_method": relation["target"], - "path": relation["path"], - "line_span": relation["line_span"], - "edge_type": relation["edge_type"], - } - for relation in relations - if _component_name(relation["source"]) and _component_name(relation["target"]) - ] - relation_rows = [row for row in relation_rows if row["source"] != row["target"]] - relation_verbs = _unique(row["verb"] for row in relation_rows if row["verb"]) - # Краткие формулировки связей для обязательного упоминания в ответе - relation_summaries = [f"{r['source']} {r['verb']} {r['target']}" for r in relation_rows[:8]] - fact_gaps: list[str] = [] - if not relation_rows: - fact_gaps.append("Concrete code edges между компонентами не подтверждены.") - if components and not relation_rows: - fact_gaps.append("Компоненты есть, но связи между ними не извлечены — не придумывай связи.") - - return { - "required_components": components[:10], - "required_relations": relation_rows[:10], - "required_relation_summaries": relation_summaries[:8], - "required_relation_verbs": relation_verbs[:8], - "required_creation_edges": [row for row in relation_rows if row["edge_type"] == "instantiates"][:4], - "required_call_edges": [row for row in relation_rows if row["edge_type"] == "calls"][:4], - "required_registration_edges": [row for row in relation_rows if row["edge_type"] == "imports"][:4], - "required_data_edges": [row for row in relation_rows if row["edge_type"] in {"reads_attr", "writes_attr"}][:4], - "fact_gaps": fact_gaps, - "forbidden_labels": list(_RETRIEVAL_LABELS), - } - - -def _trace_flow_facts(bundle: EvidenceBundle, chunks: list[CodeChunkItem], relations: list[dict[str, Any]]) -> dict[str, Any]: - target = str(bundle.resolved_target or "").strip() - sorted_relations = sorted( - relations, - key=lambda item: (item["path"], item["sort_line"], item["source"], item["target"]), - ) - flow_steps = [ - { - "step": index, - "source": relation["source"], - "verb": relation["verb"], - "target": relation["target"], - "path": relation["path"], - "line_span": relation["line_span"], - } - for index, relation in enumerate(sorted_relations, start=1) - ] - # Упорядоченные короткие формулировки шагов для обязательного отражения в ответе - ordered_step_descriptions = [ - f"{i}. {_component_name(s['source'])} {s['verb']} {_display_call_target(s['target'])}" - for i, s in enumerate(flow_steps[:8], start=1) - ] - required_calls = _unique(_display_call_target(item["target"]) for item in flow_steps) - fact_gaps: list[str] = [] - if len(flow_steps) < 2: - fact_gaps.append("Полная последовательность шагов не подтверждена; виден только частичный flow.") - if not flow_steps: - fact_gaps.append("Конкретные sequence edges для flow не подтверждены.") - if flow_steps and len(flow_steps) < 3: - fact_gaps.append("Цепочка короткая — не заявляй полноту потока.") - - return { - "required_flow_steps": flow_steps[:10], - "required_calls": required_calls[:10], - "required_sequence_edges": flow_steps[:10], - "ordered_step_descriptions": ordered_step_descriptions[:8], - "required_files": _unique(chunk.path for chunk in chunks if _chunk_matches_target(chunk, target) and chunk.path)[:4], - "fact_gaps": fact_gaps, - } - - -def _semantic_hints(chunks: list[CodeChunkItem]) -> list[dict[str, Any]]: - hints: list[dict[str, Any]] = [] - for chunk in chunks: - if chunk.layer != "C4_SEMANTIC_ROLES": - continue - meta = dict(chunk.metadata or {}) - hints.append( - { - "symbol": meta.get("symbol_name") or chunk.title, - "role": meta.get("role"), - "path": chunk.path, - "confidence": meta.get("confidence"), - } - ) - return hints[:8] - - -def _normalized_relations(relations: list[dict[str, Any]]) -> list[dict[str, Any]]: - normalized: list[dict[str, Any]] = [] - for relation in relations: - metadata = dict(relation.get("metadata") or {}) - edge_type = str(metadata.get("edge_type") or relation.get("edge_type") or "").strip() - source = _clean_endpoint(str(metadata.get("src_qname") or relation.get("source") or "").strip()) - target = _clean_endpoint(str(metadata.get("dst_ref") or relation.get("target") or "").strip()) - if not edge_type or not source or not target: - continue - normalized.append( - { - "edge_type": edge_type, - "source": source, - "target": target, - "verb": _RELATION_VERBS.get(edge_type, edge_type), - "path": str(relation.get("path") or ""), - "line_span": _line_span(relation), - "sort_line": int(relation.get("start_line") or 0), - } - ) - return normalized - - -def _signature_payload(chunk: CodeChunkItem) -> dict[str, Any]: - meta = dict(chunk.metadata or {}) - signature = str(meta.get("signature") or chunk.content or "").strip() - match = _SIGNATURE_RE.search(signature) - args = [part.strip() for part in match.group("args").split(",") if part.strip()] if match else [] - return { - "name": _clean_endpoint(str(meta.get("qname") or chunk.title or "").strip()), - "kind": str(meta.get("kind") or "").strip(), - "args": args, - } - - -def _fallback_calls(chunks: list[CodeChunkItem], target: str) -> list[str]: - values: list[str] = [] - for chunk in chunks: - if not _chunk_matches_target(chunk, target): - continue - values.extend(_display_call_target(item) for item in _CALL_RE.findall(chunk.content or "")) - return _unique(values) - - -def _chunk_matches_target(chunk: CodeChunkItem, target: str) -> bool: - if not target: - return True - title = _clean_endpoint(chunk.title) - symbol = _clean_endpoint(str(dict(chunk.metadata or {}).get("qname") or "")) - return _is_target_symbol(title, target) or _is_target_symbol(symbol, target) or target.lower() in (chunk.content or "").lower() - - -def _is_target_relation(relation: dict[str, Any], target: str) -> bool: - if not target: - return True - return _is_target_symbol(relation["source"], target) or _component_name(relation["source"]) == target or _component_name(relation["target"]) == target - - -def _is_target_symbol(name: str, target: str) -> bool: - clean_name = _clean_endpoint(name) - clean_target = _clean_endpoint(target) - if not clean_target: - return True - return clean_name == clean_target or clean_name.startswith(f"{clean_target}.") - - -def _component_name(value: str) -> str: - clean = _clean_endpoint(value) - if not clean: - return "" - parts = clean.split(".") - if len(parts) <= 1: - return clean - if len(parts) == 2: - return clean if parts[1][:1].isupper() else parts[0] - return ".".join(parts[:-1]) - - -def _short_method_name(qname: str) -> str: - """Короткое отображаемое имя метода для ответа (Class.method или method()).""" - clean = _clean_endpoint(qname) - if not clean: - return "" - parts = clean.split(".") - tail = parts[-1] - if len(parts) >= 2 and parts[-2][:1].isupper(): - return f"{parts[-2]}.{tail}()" if tail != "__init__" else f"{parts[-2]}.__init__()" - return f"{tail}()" if tail else clean - - -def _display_call_target(value: str) -> str: - clean = _clean_endpoint(value) - if not clean: - return "" - tail = clean.rsplit(".", 1)[-1] - return f"{tail}()" if tail and tail != clean else clean - - -def _display_dependency_target(value: str) -> str: - clean = _clean_endpoint(value) - if not clean: - return "" - return _component_name(clean) or clean - - -def _clean_endpoint(value: str) -> str: - clean = str(value or "").strip() - if not clean: - return "" - for label in _RETRIEVAL_LABELS: - clean = clean.replace(f".{label}", "") - clean = clean.replace(label, "") - return clean.strip(".: ") - - -def _line_span(item: dict[str, Any]) -> str: - start = item.get("start_line") - end = item.get("end_line") - if start is None and end is None: - return "?:?" - return f"{start or '?'}-{end or '?'}" - - -def _unique(values: Any) -> list[str]: - seen: set[str] = set() - result: list[str] = [] - for value in values: - text = str(value or "").strip() - if not text or text in seen: - continue - seen.add(text) - result.append(text) - return result diff --git a/src/app/modules/agent/runtime/steps/context/answer_synthesis.py b/src/app/modules/agent/runtime/steps/context/answer_synthesis.py deleted file mode 100644 index c566b58..0000000 --- a/src/app/modules/agent/runtime/steps/context/answer_synthesis.py +++ /dev/null @@ -1,71 +0,0 @@ -"""Сборка AnswerSynthesisInput из EvidenceBundle для этапа LLM.""" - -from __future__ import annotations - -from app.modules.agent.runtime.steps.context.answer_fact_curator import build_curated_answer_facts -from app.modules.agent.runtime.steps.context.contracts import AnswerSynthesisInput, EvidenceBundle - - -def build_answer_synthesis_input( - user_question: str, - bundle: EvidenceBundle, -) -> AnswerSynthesisInput: - scenario = bundle.resolved_sub_intent or "EXPLAIN" - target = bundle.resolved_target - sufficient = bundle.sufficient - curated = build_curated_answer_facts(bundle) - semantic_hints = list(curated.get("semantic_hints") or []) - - fast_lines = [ - f"Scenario: {scenario}", - f"Target: {target or 'none'}", - f"Evidence chunks: {bundle.evidence_count}", - f"Primary chunks: {curated.get('primary_chunk_count', 0)}", - f"Semantic hints: {len(semantic_hints)}", - f"Relations: {curated.get('relation_count', 0)}", - f"Sufficient: {sufficient}", - ] - if bundle.failure_reasons: - fast_lines.append(f"Failure reasons: {', '.join(bundle.failure_reasons)}") - fast_context = "\n".join(fast_lines) - - deep_parts = [] - primary_chunks = [chunk for chunk in bundle.code_chunks if scenario.upper() not in {"EXPLAIN", "ARCHITECTURE"} or chunk.layer != "C4_SEMANTIC_ROLES"] - for i, c in enumerate(primary_chunks[:30], 1): - deep_parts.append(f"[{i}] {c.path}:{c.start_line or '?'}-{c.end_line or '?'}\n{c.content[:800]}") - deep_context = "\n\n---\n\n".join(deep_parts) if deep_parts else "(no code chunks)" - - evidence_summary = [] - for c in primary_chunks[:20]: - evidence_summary.append({ - "layer": c.layer, - "path": c.path, - "title": c.title, - "lines": f"{c.start_line or '?'}-{c.end_line or '?'}", - }) - if semantic_hints: - evidence_summary.append({"kind": "semantic_hints", "count": len(semantic_hints)}) - if bundle.entrypoints: - evidence_summary.append({"kind": "entrypoints", "count": len(bundle.entrypoints)}) - if bundle.test_evidence: - evidence_summary.append({"kind": "tests", "count": len(bundle.test_evidence)}) - - hints = list(bundle.failure_reasons) if bundle.failure_reasons else [] - if not sufficient and bundle.retrieval_summary.get("missing_layers"): - hints.append(f"Missing layers: {', '.join(bundle.retrieval_summary['missing_layers'])}") - scenario_key = scenario.lower() - scenario_facts = dict(curated.get(scenario_key) or {}) - hints.extend(list(scenario_facts.get("fact_gaps") or [])) - - return AnswerSynthesisInput( - user_question=user_question, - resolved_scenario=scenario, - resolved_target=target, - fast_context=fast_context, - deep_context=deep_context, - evidence_summary=evidence_summary, - semantic_hints=semantic_hints, - curated_facts=curated, - evidence_sufficient=sufficient, - diagnostic_hints=hints, - ) diff --git a/src/app/modules/agent/runtime/steps/context/contracts.py b/src/app/modules/agent/runtime/steps/context/contracts.py deleted file mode 100644 index e8a5972..0000000 --- a/src/app/modules/agent/runtime/steps/context/contracts.py +++ /dev/null @@ -1,146 +0,0 @@ -"""Типизированные контракты пайплайна: RouterResult, RetrievalRequest, RetrievalResult, EvidenceBundle, AnswerSynthesisInput, DiagnosticsReport.""" - -from __future__ import annotations - -from typing import Any, Literal - -from pydantic import BaseModel, ConfigDict, Field - -from app.modules.agent.intent_router_v2.models import IntentRouterResult - -RouterResult = IntentRouterResult - -FailureReason = Literal[ - "router_low_confidence", - "target_not_resolved", - "path_scope_empty", - "layer_c0_empty", - "layer_c1_empty", - "layer_c2_empty", - "entrypoints_not_found", - "tests_not_found", - "insufficient_evidence", -] - -FAILURE_REASONS: tuple[FailureReason, ...] = ( - "router_low_confidence", - "target_not_resolved", - "path_scope_empty", - "layer_c0_empty", - "layer_c1_empty", - "layer_c2_empty", - "entrypoints_not_found", - "tests_not_found", - "insufficient_evidence", -) - - -class RetrievalRequest(BaseModel): - model_config = ConfigDict(extra="forbid") - - rag_session_id: str - query: str - sub_intent: str = "EXPLAIN" - path_scope: list[str] = Field(default_factory=list) - keyword_hints: list[str] = Field(default_factory=list) - symbol_candidates: list[str] = Field(default_factory=list) - requested_layers: list[str] = Field(default_factory=list) - retrieval_spec: Any = None - retrieval_constraints: Any = None - query_plan: Any = None - - -class CodeChunkItem(BaseModel): - model_config = ConfigDict(extra="forbid") - - layer: str - path: str - title: str = "" - content: str = "" - start_line: int | None = None - end_line: int | None = None - metadata: dict[str, Any] = Field(default_factory=dict) - - -class LayerOutcome(BaseModel): - model_config = ConfigDict(extra="forbid") - - layer_id: str - hit_count: int = 0 - empty: bool = True - fallback_used: bool = False - - -class RetrievalResult(BaseModel): - model_config = ConfigDict(extra="forbid") - - target_symbol_candidates: list[str] = Field(default_factory=list) - resolved_symbol: str | None = None - symbol_resolution_status: str = "not_requested" - file_candidates: list[str] = Field(default_factory=list) - code_chunks: list[CodeChunkItem] = Field(default_factory=list) - relations: list[dict[str, Any]] = Field(default_factory=list) - semantic_hints: list[dict[str, Any]] = Field(default_factory=list) - entrypoints: list[dict[str, Any]] = Field(default_factory=list) - test_candidates: list[dict[str, Any]] = Field(default_factory=list) - layer_outcomes: list[LayerOutcome] = Field(default_factory=list) - missing_layers: list[str] = Field(default_factory=list) - raw_rows: list[dict[str, Any]] = Field(default_factory=list) - retrieval_report: dict[str, Any] = Field(default_factory=dict) - - -class EvidenceBundle(BaseModel): - model_config = ConfigDict(extra="forbid") - - resolved_intent: str = "" - resolved_sub_intent: str = "" - resolved_target: str | None = None - target_type: Literal["symbol", "path", "file", "entrypoint", "none"] = "none" - target_symbol_candidates: list[str] = Field(default_factory=list) - file_candidates: list[str] = Field(default_factory=list) - code_chunks: list[CodeChunkItem] = Field(default_factory=list) - relations: list[dict[str, Any]] = Field(default_factory=list) - entrypoints: list[dict[str, Any]] = Field(default_factory=list) - test_evidence: list[dict[str, Any]] = Field(default_factory=list) - evidence_count: int = 0 - sufficient: bool = False - failure_reasons: list[str] = Field(default_factory=list) - retrieval_summary: dict[str, Any] = Field(default_factory=dict) - - -class AnswerSynthesisInput(BaseModel): - model_config = ConfigDict(extra="forbid") - - user_question: str = "" - resolved_scenario: str = "" - resolved_target: str | None = None - fast_context: str = "" - deep_context: str = "" - evidence_summary: list[dict[str, Any]] = Field(default_factory=list) - semantic_hints: list[dict[str, Any]] = Field(default_factory=list) - curated_facts: dict[str, Any] = Field(default_factory=dict) - evidence_sufficient: bool = False - diagnostic_hints: list[str] = Field(default_factory=list) - - -class DiagnosticsReport(BaseModel): - model_config = ConfigDict(extra="forbid") - - intent_correct: bool | None = None - target_found: bool = False - layers_used: list[str] = Field(default_factory=list) - retrieval_sufficient: bool = False - answer_mode: Literal["normal", "degraded", "insufficient"] = "normal" - resolved_target: str | None = None - answer_policy_branch: str = "" - decision_reason: str = "" - - router_result: dict[str, Any] = Field(default_factory=dict) - retrieval_request: dict[str, Any] = Field(default_factory=dict) - per_layer_outcome: list[dict[str, Any]] = Field(default_factory=list) - empty_layers: list[str] = Field(default_factory=list) - evidence_gate_decision: dict[str, Any] = Field(default_factory=dict) - evidence_gate_input: dict[str, Any] = Field(default_factory=dict) - post_evidence_gate: dict[str, Any] = Field(default_factory=dict) - failure_reasons: list[str] = Field(default_factory=list) - timings_ms: dict[str, int] = Field(default_factory=dict) diff --git a/src/app/modules/agent/runtime/steps/context/diagnostics.py b/src/app/modules/agent/runtime/steps/context/diagnostics.py deleted file mode 100644 index b33cb37..0000000 --- a/src/app/modules/agent/runtime/steps/context/diagnostics.py +++ /dev/null @@ -1,135 +0,0 @@ -"""Диагностика пайплайна CODE_QA: сводка уровня 1 и детали уровня 2.""" - -from __future__ import annotations - -from typing import Any - -from app.modules.agent.runtime.steps.context.contracts import ( - DiagnosticsReport, - EvidenceBundle, - RetrievalRequest, - RetrievalResult, - RouterResult, -) - - -def build_diagnostics_report( - *, - router_result: RouterResult, - retrieval_request: RetrievalRequest | None, - retrieval_result: RetrievalResult | None, - evidence_bundle: EvidenceBundle | None, - answer_mode: str = "normal", - timings_ms: dict[str, int] | None = None, - resolved_target: str | None = None, - answer_policy_branch: str = "", - decision_reason: str = "", - evidence_gate_input: dict[str, Any] | None = None, - post_evidence_gate: dict[str, Any] | None = None, -) -> DiagnosticsReport: - timings = dict(timings_ms or {}) - req = retrieval_request - res = retrieval_result - bundle = evidence_bundle - - intent_correct = None - target_found = bool(bundle and bundle.resolved_target) - layers_used = list(req.requested_layers) if req else [] - retrieval_sufficient = bool(bundle and bundle.sufficient) - if not retrieval_sufficient and bundle: - answer_mode = "degraded" if bundle.evidence_count else "insufficient" - failure_reasons = list(bundle.failure_reasons) if bundle else [] - - router_result_dict = _router_result_to_dict(router_result) - retrieval_request_dict = _retrieval_request_to_dict(req) if req else {} - per_layer = [] - if res: - for o in res.layer_outcomes: - per_layer.append({ - "layer_id": o.layer_id, - "hit_count": o.hit_count, - "empty": o.empty, - "fallback_used": o.fallback_used, - }) - empty_layers = list(res.missing_layers) if res else [] - evidence_gate_decision = {} - if bundle is not None: - evidence_gate_decision = { - "sufficient": bundle.sufficient, - "failure_reasons": list(bundle.failure_reasons), - "evidence_count": bundle.evidence_count, - } - - return DiagnosticsReport( - intent_correct=intent_correct, - target_found=target_found, - layers_used=layers_used, - retrieval_sufficient=retrieval_sufficient, - answer_mode=answer_mode, - resolved_target=resolved_target or (bundle.resolved_target if bundle else None), - answer_policy_branch=answer_policy_branch, - decision_reason=decision_reason, - router_result=router_result_dict, - retrieval_request=retrieval_request_dict, - per_layer_outcome=per_layer, - empty_layers=empty_layers, - evidence_gate_decision=evidence_gate_decision, - evidence_gate_input=dict(evidence_gate_input or {}), - post_evidence_gate=dict(post_evidence_gate or {}), - failure_reasons=failure_reasons, - timings_ms=timings, - ) - - -def build_level1_summary(report: DiagnosticsReport) -> dict[str, Any]: - return { - "intent_correct": report.intent_correct, - "target_found": report.target_found, - "layers_used": report.layers_used, - "retrieval_sufficient": report.retrieval_sufficient, - "answer_mode": report.answer_mode, - "resolved_target": report.resolved_target, - "answer_policy_branch": report.answer_policy_branch, - "decision_reason": report.decision_reason, - "failure_reasons": report.failure_reasons, - } - - -def build_level2_detail(report: DiagnosticsReport) -> dict[str, Any]: - return { - "router_result": report.router_result, - "retrieval_request": report.retrieval_request, - "per_layer_outcome": report.per_layer_outcome, - "empty_layers": report.empty_layers, - "resolved_target": report.resolved_target, - "answer_policy_branch": report.answer_policy_branch, - "decision_reason": report.decision_reason, - "evidence_gate_input": report.evidence_gate_input, - "evidence_gate_decision": report.evidence_gate_decision, - "post_evidence_gate": report.post_evidence_gate, - "failure_reasons": report.failure_reasons, - "timings_ms": report.timings_ms, - } - - -def _router_result_to_dict(r: RouterResult) -> dict[str, Any]: - return { - "intent": r.intent, - "graph_id": r.graph_id, - "conversation_mode": r.conversation_mode, - "retrieval_profile": r.retrieval_profile, - "sub_intent": r.query_plan.sub_intent if r.query_plan else None, - "path_scope": list(getattr(r.retrieval_spec.filters, "path_scope", []) or []), - "layers": [str(q.layer_id) for q in (r.retrieval_spec.layer_queries or [])], - "symbol_resolution_status": r.symbol_resolution.status if r.symbol_resolution else None, - } - - -def _retrieval_request_to_dict(req: RetrievalRequest) -> dict[str, Any]: - return { - "rag_session_id": req.rag_session_id, - "query": req.query, - "sub_intent": req.sub_intent, - "path_scope": list(req.path_scope), - "requested_layers": list(req.requested_layers), - } diff --git a/src/app/modules/agent/runtime/steps/context/evidence_bundle_builder.py b/src/app/modules/agent/runtime/steps/context/evidence_bundle_builder.py deleted file mode 100644 index 8edeff7..0000000 --- a/src/app/modules/agent/runtime/steps/context/evidence_bundle_builder.py +++ /dev/null @@ -1,57 +0,0 @@ -"""Сборка EvidenceBundle из RetrievalResult и результата роутера.""" - -from __future__ import annotations - -from app.modules.agent.runtime.steps.context.contracts import EvidenceBundle, RetrievalResult, RouterResult - - -def build_evidence_bundle( - retrieval_result: RetrievalResult, - router_result: RouterResult, -) -> EvidenceBundle: - intent = router_result.intent or "CODE_QA" - sub_intent = (router_result.query_plan and router_result.query_plan.sub_intent) or "EXPLAIN" - resolved_target: str | None = None - target_type: str = "none" - - if retrieval_result.resolved_symbol: - resolved_target = retrieval_result.resolved_symbol - target_type = "symbol" - elif retrieval_result.file_candidates: - resolved_target = retrieval_result.file_candidates[0] - target_type = "file" - if retrieval_result.entrypoints and not resolved_target: - resolved_target = retrieval_result.entrypoints[0].get("title") or retrieval_result.entrypoints[0].get("path") - target_type = "entrypoint" - - summary = { - "chunk_count": len(retrieval_result.code_chunks), - "relation_count": len(retrieval_result.relations), - "semantic_hint_count": len(retrieval_result.semantic_hints), - "entrypoint_count": len(retrieval_result.entrypoints), - "test_candidate_count": len(retrieval_result.test_candidates), - "layers_with_hits": [ - o.layer_id for o in retrieval_result.layer_outcomes if o.hit_count > 0 - ], - "missing_layers": list(retrieval_result.missing_layers), - } - - file_candidates = list(retrieval_result.file_candidates) - target_symbols = list(retrieval_result.target_symbol_candidates) - - return EvidenceBundle( - resolved_intent=intent, - resolved_sub_intent=sub_intent, - resolved_target=resolved_target, - target_type=target_type, - target_symbol_candidates=target_symbols, - file_candidates=file_candidates, - code_chunks=list(retrieval_result.code_chunks), - relations=list(retrieval_result.relations), - entrypoints=list(retrieval_result.entrypoints), - test_evidence=list(retrieval_result.test_candidates), - evidence_count=len(retrieval_result.code_chunks), - sufficient=False, - failure_reasons=[], - retrieval_summary=summary, - ) diff --git a/src/app/modules/agent/runtime/steps/context/retrieval_request_builder.py b/src/app/modules/agent/runtime/steps/context/retrieval_request_builder.py deleted file mode 100644 index 05b8835..0000000 --- a/src/app/modules/agent/runtime/steps/context/retrieval_request_builder.py +++ /dev/null @@ -1,24 +0,0 @@ -"""Сборка RetrievalRequest из RouterResult для пайплайна CODE_QA.""" - -from __future__ import annotations - -from app.modules.agent.runtime.steps.context.contracts import RetrievalRequest, RouterResult - - -def build_retrieval_request(router_result: RouterResult, rag_session_id: str) -> RetrievalRequest: - query_plan = router_result.query_plan - spec = router_result.retrieval_spec - path_scope = list(getattr(spec.filters, "path_scope", []) or []) - layers = [str(q.layer_id) for q in (spec.layer_queries or [])] - return RetrievalRequest( - rag_session_id=rag_session_id, - query=query_plan.normalized or query_plan.raw or "", - sub_intent=query_plan.sub_intent or "EXPLAIN", - path_scope=path_scope, - keyword_hints=list(query_plan.keyword_hints or []), - symbol_candidates=list(query_plan.symbol_candidates or []), - requested_layers=layers, - retrieval_spec=spec, - retrieval_constraints=router_result.retrieval_constraints, - query_plan=query_plan, - ) diff --git a/src/app/modules/agent/runtime/steps/context/retrieval_result_builder.py b/src/app/modules/agent/runtime/steps/context/retrieval_result_builder.py deleted file mode 100644 index 7a4e1cd..0000000 --- a/src/app/modules/agent/runtime/steps/context/retrieval_result_builder.py +++ /dev/null @@ -1,241 +0,0 @@ -"""Сборка нормализованного RetrievalResult из сырых строк retrieval и отчёта.""" - -from __future__ import annotations - -import re - -from app.modules.agent.runtime.steps.context.contracts import CodeChunkItem, LayerOutcome, RetrievalResult -from app.modules.rag.retrieval.test_filter import is_test_path - -_ROUTE_RE = re.compile(r'@[\w\.]+\.(get|post|put|delete|patch|options|head)\(\s*["\']([^"\']+)["\']') -_DEF_RE = re.compile(r"async\s+def\s+(\w+)|def\s+(\w+)") - - -def build_retrieval_result( - raw_rows: list[dict], - retrieval_report: dict | None, - symbol_resolution: dict | None, -) -> RetrievalResult: - report = retrieval_report or {} - sym = symbol_resolution or {} - layers_seen: set[str] = set() - code_chunks: list[CodeChunkItem] = [] - relations: list[dict] = [] - semantic_hints: list[dict] = [] - entrypoint_rows: list[CodeChunkItem] = [] - test_candidates: list[dict] = [] - file_candidates: list[str] = [] - target_symbols: list[str] = [] - - for row in raw_rows: - chunk = _to_chunk(row) - layer = chunk.layer - path = chunk.path - if layer: - layers_seen.add(layer) - if path and path not in file_candidates: - file_candidates.append(path) - code_chunks.append(chunk) - - if layer == "C1_SYMBOL_CATALOG": - title = chunk.title.strip() - if title and title not in target_symbols: - target_symbols.append(title) - continue - if layer == "C2_DEPENDENCY_GRAPH": - relations.append(_relation_payload(chunk)) - continue - if layer == "C4_SEMANTIC_ROLES": - semantic_hints.append(_semantic_hint_payload(chunk)) - continue - if layer == "C3_ENTRYPOINTS": - entrypoint_rows.append(chunk) - continue - if is_test_path(path): - test_candidates.append({"path": path, "title": chunk.title, "content": chunk.content[:300]}) - - entrypoints = _normalize_entrypoints(entrypoint_rows, code_chunks) - executed = list(report.get("executed_layers") or []) - missing_layers = [lid for lid in executed if lid not in layers_seen] - layer_outcomes = _layer_outcomes(raw_rows, executed, layers_seen, report) - - resolved = str(sym.get("resolved_symbol") or "").strip() or None - status = str(sym.get("status") or "not_requested") - - return RetrievalResult( - target_symbol_candidates=target_symbols, - resolved_symbol=resolved, - symbol_resolution_status=status, - file_candidates=file_candidates, - code_chunks=code_chunks, - relations=relations, - semantic_hints=semantic_hints, - entrypoints=entrypoints, - test_candidates=test_candidates, - layer_outcomes=layer_outcomes, - missing_layers=missing_layers, - raw_rows=raw_rows, - retrieval_report=report, - ) - - -def _to_chunk(row: dict) -> CodeChunkItem: - return CodeChunkItem( - layer=str(row.get("layer") or ""), - path=str(row.get("path") or ""), - title=str(row.get("title") or ""), - content=str(row.get("content") or ""), - start_line=row.get("span_start"), - end_line=row.get("span_end"), - metadata=dict(row.get("metadata") or {}), - ) - - -def _normalize_entrypoints(entrypoint_rows: list[CodeChunkItem], code_chunks: list[CodeChunkItem]) -> list[dict]: - chunks_by_path: dict[str, list[CodeChunkItem]] = {} - for chunk in code_chunks: - chunks_by_path.setdefault(chunk.path, []).append(chunk) - - normalized: list[dict] = [] - seen: set[tuple[str, str, str, int | None, int | None]] = set() - for chunk in entrypoint_rows: - item = _entrypoint_payload(chunk, chunks_by_path.get(chunk.path, [])) - key = ( - str(item.get("http_method") or ""), - str(item.get("route_path") or ""), - str(item.get("path") or ""), - chunk.start_line, - chunk.end_line, - ) - if key in seen: - continue - seen.add(key) - normalized.append(item) - return sorted(normalized, key=_entrypoint_sort_key) - - -def _relation_payload(chunk: CodeChunkItem) -> dict: - metadata = dict(chunk.metadata or {}) - edge_type = str(metadata.get("edge_type") or "").strip() - src_qname = str(metadata.get("src_qname") or "").strip() - dst_ref = str(metadata.get("dst_ref") or "").strip() - return { - "path": chunk.path, - "start_line": chunk.start_line, - "end_line": chunk.end_line, - "edge_type": edge_type, - "source": src_qname, - "target": dst_ref, - "source_component": _component_name(src_qname), - "target_component": _component_name(dst_ref), - "has_retrieval_label": "dataflow_slice" in dst_ref or "dataflow_slice" in src_qname, - "metadata": metadata, - "content": chunk.content[:500], - } - - -def _semantic_hint_payload(chunk: CodeChunkItem) -> dict: - metadata = dict(chunk.metadata or {}) - return { - "path": chunk.path, - "title": chunk.title, - "symbol": metadata.get("symbol_name") or chunk.title, - "role": metadata.get("role"), - "confidence": metadata.get("confidence"), - "content": chunk.content[:300], - } - - -def _component_name(value: str) -> str: - cleaned = str(value or "").replace(".dataflow_slice", "").strip(". ") - if "." not in cleaned: - return cleaned - head, tail = cleaned.rsplit(".", 1) - return head if tail and not tail[:1].isupper() else cleaned - - -def _entrypoint_payload(chunk: CodeChunkItem, siblings: list[CodeChunkItem]) -> dict: - methods = [str(item).strip().upper() for item in list(dict(chunk.metadata).get("lang_payload", {}).get("methods", [])) if str(item).strip()] - method = methods[0] if methods else "" - route_path = _infer_route_path(chunk, siblings) - handler_symbol = _infer_handler_symbol(chunk, siblings) - return { - "path": chunk.path, - "title": chunk.title, - "start_line": chunk.start_line, - "end_line": chunk.end_line, - "http_method": method, - "route_path": route_path, - "handler_symbol": handler_symbol, - "handler_symbol_id": dict(chunk.metadata).get("handler_symbol_id"), - "route_or_command": dict(chunk.metadata).get("route_or_command"), - "metadata": chunk.metadata, - } - - -def _infer_route_path(chunk: CodeChunkItem, siblings: list[CodeChunkItem]) -> str: - for candidate in siblings: - if candidate.layer != "C0_SOURCE_CHUNKS": - continue - match = _ROUTE_RE.search(_window_text(candidate, chunk.start_line)) - if match: - return match.group(2).strip() - return "" - - -def _infer_handler_symbol(chunk: CodeChunkItem, siblings: list[CodeChunkItem]) -> str: - for candidate in siblings: - if candidate.layer != "C0_SOURCE_CHUNKS": - continue - match = _DEF_RE.search(_window_text(candidate, chunk.start_line, before=2, after=12)) - if match: - return match.group(1) or match.group(2) or "" - return "" - - -def _window_text(chunk: CodeChunkItem, target_line: int | None, *, before: int = 6, after: int = 8) -> str: - text = chunk.content or "" - if not text or not target_line or not chunk.start_line: - return text - lines = text.splitlines() - offset = max(0, target_line - chunk.start_line - before) - end = min(len(lines), target_line - chunk.start_line + after) - if offset >= end: - return text - return "\n".join(lines[offset:end]) - - -def _entrypoint_sort_key(item: dict) -> tuple[int, int, str, str, int]: - route_path = str(item.get("route_path") or "") - method = str(item.get("http_method") or "") - score = 0 - if route_path: - score += 3 - if route_path and route_path != "/": - score += 3 - if method: - score += 1 - return (-score, len(route_path or "zzzz"), method, str(item.get("path") or ""), int(item.get("start_line") or 0)) - - -def _layer_outcomes(raw_rows: list[dict], executed: list[str], layers_seen: set[str], report: dict) -> list[LayerOutcome]: - layer_outcomes = [ - LayerOutcome( - layer_id=layer_id, - hit_count=sum(1 for r in raw_rows if str(r.get("layer") or "") == layer_id), - empty=layer_id not in layers_seen, - fallback_used=bool((report.get("fallback") or {}).get("used") and report.get("retrieval_mode_by_layer", {}).get(layer_id)), - ) - for layer_id in executed - ] - if not executed and raw_rows: - for layer in layers_seen: - layer_outcomes.append( - LayerOutcome( - layer_id=layer, - hit_count=sum(1 for r in raw_rows if str(r.get("layer") or "") == layer), - empty=False, - fallback_used=False, - ) - ) - return layer_outcomes diff --git a/src/app/modules/agent/runtime/steps/explain/__init__.py b/src/app/modules/agent/runtime/steps/explain/__init__.py deleted file mode 100644 index 59bf6a9..0000000 --- a/src/app/modules/agent/runtime/steps/explain/__init__.py +++ /dev/null @@ -1,36 +0,0 @@ -from __future__ import annotations - -from importlib import import_module - -__all__ = [ - "CodeExcerpt", - "CodeExplainRetrieverV2", - "CodeGraphRepository", - "EvidenceItem", - "ExplainIntent", - "ExplainIntentBuilder", - "ExplainPack", - "LayeredRetrievalGateway", - "PromptBudgeter", - "TracePath", -] - - -def __getattr__(name: str): - module_map = { - "CodeExcerpt": "app.modules.agent.runtime.steps.explain.models", - "EvidenceItem": "app.modules.agent.runtime.steps.explain.models", - "ExplainIntent": "app.modules.agent.runtime.steps.explain.models", - "ExplainPack": "app.modules.agent.runtime.steps.explain.models", - "TracePath": "app.modules.agent.runtime.steps.explain.models", - "ExplainIntentBuilder": "app.modules.agent.runtime.steps.explain.intent_builder", - "PromptBudgeter": "app.modules.agent.runtime.steps.explain.budgeter", - "LayeredRetrievalGateway": "app.modules.agent.runtime.steps.explain.layered_gateway", - "CodeGraphRepository": "app.modules.agent.runtime.steps.explain.graph_repository", - "CodeExplainRetrieverV2": "app.modules.agent.runtime.steps.explain.retriever_v2", - } - module_name = module_map.get(name) - if module_name is None: - raise AttributeError(name) - module = import_module(module_name) - return getattr(module, name) diff --git a/src/app/modules/agent/runtime/steps/explain/budgeter.py b/src/app/modules/agent/runtime/steps/explain/budgeter.py deleted file mode 100644 index 4f925b5..0000000 --- a/src/app/modules/agent/runtime/steps/explain/budgeter.py +++ /dev/null @@ -1,62 +0,0 @@ -from __future__ import annotations - -import json - -from app.modules.agent.runtime.steps.explain.models import ExplainPack - - -class PromptBudgeter: - def __init__( - self, - *, - max_paths: int = 3, - max_symbols: int = 25, - max_excerpts: int = 40, - max_chars: int = 30000, - ) -> None: - self._max_paths = max_paths - self._max_symbols = max_symbols - self._max_excerpts = max_excerpts - self._max_chars = max_chars - - def build_prompt_input(self, question: str, pack: ExplainPack) -> str: - symbol_ids: list[str] = [] - for path in pack.trace_paths[: self._max_paths]: - for symbol_id in path.symbol_ids: - if symbol_id and symbol_id not in symbol_ids and len(symbol_ids) < self._max_symbols: - symbol_ids.append(symbol_id) - excerpts = [] - total_chars = 0 - for excerpt in pack.code_excerpts: - if symbol_ids and excerpt.symbol_id and excerpt.symbol_id not in symbol_ids: - continue - body = excerpt.content.strip() - remaining = self._max_chars - total_chars - if remaining <= 0 or len(excerpts) >= self._max_excerpts: - break - if len(body) > remaining: - body = body[:remaining].rstrip() + "...[truncated]" - excerpts.append( - { - "evidence_id": excerpt.evidence_id, - "title": excerpt.title, - "path": excerpt.path, - "start_line": excerpt.start_line, - "end_line": excerpt.end_line, - "focus": excerpt.focus, - "content": body, - } - ) - total_chars += len(body) - payload = { - "question": question, - "intent": pack.intent.model_dump(mode="json"), - "selected_entrypoints": [item.model_dump(mode="json") for item in pack.selected_entrypoints[:5]], - "seed_symbols": [item.model_dump(mode="json") for item in pack.seed_symbols[: self._max_symbols]], - "trace_paths": [path.model_dump(mode="json") for path in pack.trace_paths[: self._max_paths]], - "evidence_index": {key: value.model_dump(mode="json") for key, value in pack.evidence_index.items()}, - "code_excerpts": excerpts, - "missing": pack.missing, - "conflicts": pack.conflicts, - } - return json.dumps(payload, ensure_ascii=False, indent=2) diff --git a/src/app/modules/agent/runtime/steps/explain/excerpt_planner.py b/src/app/modules/agent/runtime/steps/explain/excerpt_planner.py deleted file mode 100644 index 406073e..0000000 --- a/src/app/modules/agent/runtime/steps/explain/excerpt_planner.py +++ /dev/null @@ -1,59 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.steps.explain.models import CodeExcerpt, LayeredRetrievalItem - - -class ExcerptPlanner: - _FOCUS_TOKENS = ("raise", "except", "db", "select", "insert", "update", "delete", "http", "publish", "emit") - - def plan(self, chunk: LayeredRetrievalItem, *, evidence_id: str, symbol_id: str | None) -> list[CodeExcerpt]: - location = chunk.location - if location is None: - return [] - excerpts = [ - CodeExcerpt( - evidence_id=evidence_id, - symbol_id=symbol_id, - title=chunk.title, - path=location.path, - start_line=location.start_line, - end_line=location.end_line, - content=chunk.content.strip(), - focus="overview", - ) - ] - focus = self._focus_excerpt(chunk, evidence_id=evidence_id, symbol_id=symbol_id) - if focus is not None: - excerpts.append(focus) - return excerpts - - def _focus_excerpt( - self, - chunk: LayeredRetrievalItem, - *, - evidence_id: str, - symbol_id: str | None, - ) -> CodeExcerpt | None: - location = chunk.location - if location is None: - return None - lines = chunk.content.splitlines() - for index, line in enumerate(lines): - lowered = line.lower() - if not any(token in lowered for token in self._FOCUS_TOKENS): - continue - start = max(0, index - 2) - end = min(len(lines), index + 3) - if end - start >= len(lines): - return None - return CodeExcerpt( - evidence_id=evidence_id, - symbol_id=symbol_id, - title=f"{chunk.title}:focus", - path=location.path, - start_line=(location.start_line or 1) + start, - end_line=(location.start_line or 1) + end - 1, - content="\n".join(lines[start:end]).strip(), - focus="focus", - ) - return None diff --git a/src/app/modules/agent/runtime/steps/explain/graph_repository.py b/src/app/modules/agent/runtime/steps/explain/graph_repository.py deleted file mode 100644 index 6622e7e..0000000 --- a/src/app/modules/agent/runtime/steps/explain/graph_repository.py +++ /dev/null @@ -1,216 +0,0 @@ -from __future__ import annotations - -import json - -from sqlalchemy import text - -from app.modules.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem -from app.modules.shared.db import get_engine - - -class CodeGraphRepository: - def get_out_edges( - self, - rag_session_id: str, - src_symbol_ids: list[str], - edge_types: list[str], - limit_per_src: int, - ) -> list[LayeredRetrievalItem]: - if not src_symbol_ids: - return [] - sql = """ - SELECT path, content, layer, title, metadata_json, span_start, span_end - FROM rag_chunks - WHERE rag_session_id = :sid - AND layer = 'C2_DEPENDENCY_GRAPH' - AND CAST(metadata_json AS jsonb)->>'src_symbol_id' = ANY(:src_ids) - AND CAST(metadata_json AS jsonb)->>'edge_type' = ANY(:edge_types) - ORDER BY path, span_start - """ - with get_engine().connect() as conn: - rows = conn.execute( - text(sql), - {"sid": rag_session_id, "src_ids": src_symbol_ids, "edge_types": edge_types}, - ).mappings().fetchall() - grouped: dict[str, int] = {} - items: list[LayeredRetrievalItem] = [] - for row in rows: - metadata = self._loads(row.get("metadata_json")) - src_symbol_id = str(metadata.get("src_symbol_id") or "") - grouped[src_symbol_id] = grouped.get(src_symbol_id, 0) + 1 - if grouped[src_symbol_id] > limit_per_src: - continue - items.append(self._to_item(row, metadata)) - return items - - def get_in_edges( - self, - rag_session_id: str, - dst_symbol_ids: list[str], - edge_types: list[str], - limit_per_dst: int, - ) -> list[LayeredRetrievalItem]: - if not dst_symbol_ids: - return [] - sql = """ - SELECT path, content, layer, title, metadata_json, span_start, span_end - FROM rag_chunks - WHERE rag_session_id = :sid - AND layer = 'C2_DEPENDENCY_GRAPH' - AND CAST(metadata_json AS jsonb)->>'dst_symbol_id' = ANY(:dst_ids) - AND CAST(metadata_json AS jsonb)->>'edge_type' = ANY(:edge_types) - ORDER BY path, span_start - """ - with get_engine().connect() as conn: - rows = conn.execute( - text(sql), - {"sid": rag_session_id, "dst_ids": dst_symbol_ids, "edge_types": edge_types}, - ).mappings().fetchall() - grouped: dict[str, int] = {} - items: list[LayeredRetrievalItem] = [] - for row in rows: - metadata = self._loads(row.get("metadata_json")) - dst_symbol_id = str(metadata.get("dst_symbol_id") or "") - grouped[dst_symbol_id] = grouped.get(dst_symbol_id, 0) + 1 - if grouped[dst_symbol_id] > limit_per_dst: - continue - items.append(self._to_item(row, metadata)) - return items - - def resolve_symbol_by_ref( - self, - rag_session_id: str, - dst_ref: str, - package_hint: str | None = None, - ) -> LayeredRetrievalItem | None: - ref = (dst_ref or "").strip() - if not ref: - return None - with get_engine().connect() as conn: - rows = conn.execute( - text( - """ - SELECT path, content, layer, title, metadata_json, span_start, span_end, qname - FROM rag_chunks - WHERE rag_session_id = :sid - AND layer = 'C1_SYMBOL_CATALOG' - AND (qname = :ref OR title = :ref OR qname LIKE :tail) - ORDER BY path - LIMIT 12 - """ - ), - {"sid": rag_session_id, "ref": ref, "tail": f"%{ref}"}, - ).mappings().fetchall() - best: LayeredRetrievalItem | None = None - best_score = -1 - for row in rows: - metadata = self._loads(row.get("metadata_json")) - package = str(metadata.get("package_or_module") or "") - score = 0 - if str(row.get("qname") or "") == ref: - score += 3 - if str(row.get("title") or "") == ref: - score += 2 - if package_hint and package.startswith(package_hint): - score += 3 - if package_hint and package_hint in str(row.get("path") or ""): - score += 1 - if score > best_score: - best = self._to_item(row, metadata) - best_score = score - return best - - def get_symbols_by_ids(self, rag_session_id: str, symbol_ids: list[str]) -> list[LayeredRetrievalItem]: - if not symbol_ids: - return [] - with get_engine().connect() as conn: - rows = conn.execute( - text( - """ - SELECT path, content, layer, title, metadata_json, span_start, span_end - FROM rag_chunks - WHERE rag_session_id = :sid - AND layer = 'C1_SYMBOL_CATALOG' - AND symbol_id = ANY(:symbol_ids) - ORDER BY path, span_start - """ - ), - {"sid": rag_session_id, "symbol_ids": symbol_ids}, - ).mappings().fetchall() - return [self._to_item(row, self._loads(row.get("metadata_json"))) for row in rows] - - def get_chunks_by_symbol_ids( - self, - rag_session_id: str, - symbol_ids: list[str], - prefer_chunk_type: str = "symbol_block", - ) -> list[LayeredRetrievalItem]: - symbols = self.get_symbols_by_ids(rag_session_id, symbol_ids) - chunks: list[LayeredRetrievalItem] = [] - for symbol in symbols: - location = symbol.location - if location is None: - continue - chunk = self._chunk_for_symbol(rag_session_id, symbol, prefer_chunk_type=prefer_chunk_type) - if chunk is not None: - chunks.append(chunk) - return chunks - - def _chunk_for_symbol( - self, - rag_session_id: str, - symbol: LayeredRetrievalItem, - *, - prefer_chunk_type: str, - ) -> LayeredRetrievalItem | None: - location = symbol.location - if location is None: - return None - with get_engine().connect() as conn: - rows = conn.execute( - text( - """ - SELECT path, content, layer, title, metadata_json, span_start, span_end - FROM rag_chunks - WHERE rag_session_id = :sid - AND layer = 'C0_SOURCE_CHUNKS' - AND path = :path - AND COALESCE(span_start, 0) <= :end_line - AND COALESCE(span_end, 999999) >= :start_line - ORDER BY - CASE WHEN CAST(metadata_json AS jsonb)->>'chunk_type' = :prefer_chunk_type THEN 0 ELSE 1 END, - ABS(COALESCE(span_start, 0) - :start_line) - LIMIT 1 - """ - ), - { - "sid": rag_session_id, - "path": location.path, - "start_line": location.start_line or 0, - "end_line": location.end_line or 999999, - "prefer_chunk_type": prefer_chunk_type, - }, - ).mappings().fetchall() - if not rows: - return None - row = rows[0] - return self._to_item(row, self._loads(row.get("metadata_json"))) - - def _to_item(self, row, metadata: dict) -> LayeredRetrievalItem: - return LayeredRetrievalItem( - source=str(row.get("path") or ""), - content=str(row.get("content") or ""), - layer=str(row.get("layer") or ""), - title=str(row.get("title") or ""), - metadata=metadata, - location=CodeLocation( - path=str(row.get("path") or ""), - start_line=row.get("span_start"), - end_line=row.get("span_end"), - ), - ) - - def _loads(self, value) -> dict: - if not value: - return {} - return json.loads(str(value)) diff --git a/src/app/modules/agent/runtime/steps/explain/intent_builder.py b/src/app/modules/agent/runtime/steps/explain/intent_builder.py deleted file mode 100644 index 79fb59a..0000000 --- a/src/app/modules/agent/runtime/steps/explain/intent_builder.py +++ /dev/null @@ -1,102 +0,0 @@ -from __future__ import annotations - -import re - -from app.modules.agent.runtime.steps.explain.models import ExplainHints, ExplainIntent -from app.modules.rag.retrieval.query_terms import extract_query_terms - - -class ExplainIntentBuilder: - _ROUTE_RE = re.compile(r"(/[A-Za-z0-9_./{}:-]+)") - _FILE_RE = re.compile(r"([A-Za-z0-9_./-]+\.py)") - _SYMBOL_RE = re.compile(r"\b([A-Z][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*|[A-Z][A-Za-z0-9_]{2,}|[a-z_][A-Za-z0-9_]{2,})\b") - _COMMAND_RE = re.compile(r"`([A-Za-z0-9:_-]+)`") - _TEST_KEYWORDS = ( - "тест", - "tests", - "test ", - "unit-test", - "unit test", - "юнит-тест", - "pytest", - "spec", - "как покрыто тестами", - "как проверяется", - "how is it tested", - "how it's tested", - ) - - def build(self, user_query: str) -> ExplainIntent: - normalized = " ".join((user_query or "").split()) - lowered = normalized.lower() - keywords = self._keywords(normalized) - hints = ExplainHints( - paths=self._dedupe(self._FILE_RE.findall(normalized)), - symbols=self._symbols(normalized), - endpoints=self._dedupe(self._ROUTE_RE.findall(normalized)), - commands=self._commands(normalized, lowered), - ) - return ExplainIntent( - raw_query=user_query, - normalized_query=normalized, - keywords=keywords[:12], - hints=hints, - include_tests=self._include_tests(lowered), - expected_entry_types=self._entry_types(lowered, hints), - depth=self._depth(lowered), - ) - - def _keywords(self, text: str) -> list[str]: - keywords = extract_query_terms(text) - for token in self._symbols(text): - if token not in keywords: - keywords.append(token) - for token in self._ROUTE_RE.findall(text): - if token not in keywords: - keywords.append(token) - return self._dedupe(keywords) - - def _symbols(self, text: str) -> list[str]: - values = [] - for raw in self._SYMBOL_RE.findall(text): - token = raw.strip() - if len(token) < 3: - continue - if token.endswith(".py"): - continue - values.append(token) - return self._dedupe(values) - - def _commands(self, text: str, lowered: str) -> list[str]: - values = list(self._COMMAND_RE.findall(text)) - if " command " in f" {lowered} ": - values.extend(re.findall(r"command\s+([A-Za-z0-9:_-]+)", lowered)) - if " cli " in f" {lowered} ": - values.extend(re.findall(r"cli\s+([A-Za-z0-9:_-]+)", lowered)) - return self._dedupe(values) - - def _entry_types(self, lowered: str, hints: ExplainHints) -> list[str]: - if hints.endpoints or any(token in lowered for token in ("endpoint", "route", "handler", "http", "api")): - return ["http"] - if hints.commands or any(token in lowered for token in ("cli", "command", "click", "typer")): - return ["cli"] - return ["http", "cli"] - - def _depth(self, lowered: str) -> str: - if any(token in lowered for token in ("deep", "подроб", "деталь", "full flow", "trace")): - return "deep" - if any(token in lowered for token in ("high level", "overview", "кратко", "summary")): - return "high" - return "medium" - - def _include_tests(self, lowered: str) -> bool: - normalized = f" {lowered} " - return any(token in normalized for token in self._TEST_KEYWORDS) - - def _dedupe(self, values: list[str]) -> list[str]: - result: list[str] = [] - for value in values: - item = value.strip() - if item and item not in result: - result.append(item) - return result diff --git a/src/app/modules/agent/runtime/steps/explain/layered_gateway.py b/src/app/modules/agent/runtime/steps/explain/layered_gateway.py deleted file mode 100644 index 79ca52d..0000000 --- a/src/app/modules/agent/runtime/steps/explain/layered_gateway.py +++ /dev/null @@ -1,289 +0,0 @@ -from __future__ import annotations - -import logging -from dataclasses import dataclass, field -from typing import TYPE_CHECKING, Callable - -from app.modules.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem -from app.modules.rag.retrieval.test_filter import build_test_filters, debug_disable_test_filter - -LOGGER = logging.getLogger(__name__) - -if TYPE_CHECKING: - from app.modules.rag.persistence.repository import RagRepository - from app.modules.rag.embedding.gigachat_embedder import GigaChatEmbedder - - -@dataclass(slots=True) -class LayerRetrievalResult: - items: list[LayeredRetrievalItem] - missing: list[str] = field(default_factory=list) - - -class LayeredRetrievalGateway: - def __init__(self, repository: RagRepository, embedder: GigaChatEmbedder) -> None: - self._repository = repository - self._embedder = embedder - - def retrieve_layer( - self, - rag_session_id: str, - query: str, - layer: str, - *, - limit: int, - path_prefixes: list[str] | None = None, - exclude_tests: bool = True, - prefer_non_tests: bool = False, - include_spans: bool = False, - ) -> LayerRetrievalResult: - effective_exclude_tests = exclude_tests and not debug_disable_test_filter() - filter_args = self._filter_args(effective_exclude_tests) - query_embedding: list[float] | None = None - try: - query_embedding = self._embedder.embed([query])[0] - rows = self._repository.retrieve( - rag_session_id, - query_embedding, - query_text=query, - limit=limit, - layers=[layer], - path_prefixes=path_prefixes, - exclude_path_prefixes=filter_args["exclude_path_prefixes"], - exclude_like_patterns=filter_args["exclude_like_patterns"], - prefer_non_tests=prefer_non_tests or not effective_exclude_tests, - ) - return self._success_result( - rows, - rag_session_id=rag_session_id, - label="layered retrieval", - include_spans=include_spans, - layer=layer, - exclude_tests=effective_exclude_tests, - path_prefixes=path_prefixes, - ) - except Exception as exc: - if query_embedding is None: - self._log_failure( - label="layered retrieval", - rag_session_id=rag_session_id, - layer=layer, - exclude_tests=effective_exclude_tests, - path_prefixes=path_prefixes, - exc=exc, - ) - return LayerRetrievalResult(items=[], missing=[self._failure_missing(f"layer:{layer} retrieval_failed", exc)]) - retry_result = self._retry_without_test_filter( - operation=lambda: self._repository.retrieve( - rag_session_id, - query_embedding, - query_text=query, - limit=limit, - layers=[layer], - path_prefixes=path_prefixes, - exclude_path_prefixes=None, - exclude_like_patterns=None, - prefer_non_tests=True, - ), - label="layered retrieval", - rag_session_id=rag_session_id, - include_spans=include_spans, - layer=layer, - exclude_tests=effective_exclude_tests, - path_prefixes=path_prefixes, - exc=exc, - missing_prefix=f"layer:{layer} retrieval_failed", - ) - if retry_result is not None: - return retry_result - return LayerRetrievalResult(items=[], missing=[self._failure_missing(f"layer:{layer} retrieval_failed", exc)]) - - def retrieve_lexical_code( - self, - rag_session_id: str, - query: str, - *, - limit: int, - path_prefixes: list[str] | None = None, - exclude_tests: bool = True, - include_spans: bool = False, - ) -> LayerRetrievalResult: - effective_exclude_tests = exclude_tests and not debug_disable_test_filter() - filter_args = self._filter_args(effective_exclude_tests) - try: - rows = self._repository.retrieve_lexical_code( - rag_session_id, - query_text=query, - limit=limit, - path_prefixes=path_prefixes, - exclude_path_prefixes=filter_args["exclude_path_prefixes"], - exclude_like_patterns=filter_args["exclude_like_patterns"], - prefer_non_tests=not effective_exclude_tests, - ) - return self._success_result( - rows, - rag_session_id=rag_session_id, - label="lexical retrieval", - include_spans=include_spans, - exclude_tests=effective_exclude_tests, - path_prefixes=path_prefixes, - ) - except Exception as exc: - retry_result = self._retry_without_test_filter( - operation=lambda: self._repository.retrieve_lexical_code( - rag_session_id, - query_text=query, - limit=limit, - path_prefixes=path_prefixes, - exclude_path_prefixes=None, - exclude_like_patterns=None, - prefer_non_tests=True, - ), - label="lexical retrieval", - rag_session_id=rag_session_id, - include_spans=include_spans, - exclude_tests=effective_exclude_tests, - path_prefixes=path_prefixes, - exc=exc, - missing_prefix="layer:C0 lexical_retrieval_failed", - ) - if retry_result is not None: - return retry_result - return LayerRetrievalResult(items=[], missing=[self._failure_missing("layer:C0 lexical_retrieval_failed", exc)]) - - def _retry_without_test_filter( - self, - *, - operation: Callable[[], list[dict]], - label: str, - rag_session_id: str, - include_spans: bool, - exclude_tests: bool, - path_prefixes: list[str] | None, - exc: Exception, - missing_prefix: str, - layer: str | None = None, - ) -> LayerRetrievalResult | None: - if not exclude_tests: - self._log_failure( - label=label, - rag_session_id=rag_session_id, - layer=layer, - exclude_tests=exclude_tests, - path_prefixes=path_prefixes, - exc=exc, - ) - return None - self._log_failure( - label=label, - rag_session_id=rag_session_id, - layer=layer, - exclude_tests=exclude_tests, - path_prefixes=path_prefixes, - exc=exc, - retried_without_test_filter=True, - ) - try: - rows = operation() - except Exception as retry_exc: - self._log_failure( - label=f"{label} retry", - rag_session_id=rag_session_id, - layer=layer, - exclude_tests=False, - path_prefixes=path_prefixes, - exc=retry_exc, - ) - return None - result = self._success_result( - rows, - rag_session_id=rag_session_id, - label=f"{label} retry", - include_spans=include_spans, - layer=layer, - exclude_tests=False, - path_prefixes=path_prefixes, - ) - result.missing.append(f"{missing_prefix}:retried_without_test_filter") - return result - - def _success_result( - self, - rows: list[dict], - *, - rag_session_id: str, - label: str, - include_spans: bool, - exclude_tests: bool, - path_prefixes: list[str] | None, - layer: str | None = None, - ) -> LayerRetrievalResult: - items = [self._to_item(row, include_spans=include_spans) for row in rows] - LOGGER.warning( - "%s: rag_session_id=%s layer=%s exclude_tests=%s path_prefixes=%s returned_count=%s top_paths=%s", - label, - rag_session_id, - layer, - exclude_tests, - path_prefixes or [], - len(items), - [item.source for item in items[:3]], - ) - return LayerRetrievalResult(items=items) - - def _log_failure( - self, - *, - label: str, - rag_session_id: str, - exclude_tests: bool, - path_prefixes: list[str] | None, - exc: Exception, - layer: str | None = None, - retried_without_test_filter: bool = False, - ) -> None: - LOGGER.warning( - "%s failed: rag_session_id=%s layer=%s exclude_tests=%s path_prefixes=%s retried_without_test_filter=%s error=%s", - label, - rag_session_id, - layer, - exclude_tests, - path_prefixes or [], - retried_without_test_filter, - self._exception_summary(exc), - exc_info=True, - ) - - def _filter_args(self, exclude_tests: bool) -> dict[str, list[str] | None]: - test_filters = build_test_filters() if exclude_tests else None - return { - "exclude_path_prefixes": test_filters.exclude_path_prefixes if test_filters else None, - "exclude_like_patterns": test_filters.exclude_like_patterns if test_filters else None, - } - - def _failure_missing(self, prefix: str, exc: Exception) -> str: - return f"{prefix}:{self._exception_summary(exc)}" - - def _exception_summary(self, exc: Exception) -> str: - message = " ".join(str(exc).split()) - if len(message) > 180: - message = message[:177] + "..." - return f"{type(exc).__name__}:{message or 'no_message'}" - - def _to_item(self, row: dict, *, include_spans: bool) -> LayeredRetrievalItem: - location = None - if include_spans: - location = CodeLocation( - path=str(row.get("path") or ""), - start_line=row.get("span_start"), - end_line=row.get("span_end"), - ) - return LayeredRetrievalItem( - source=str(row.get("path") or ""), - content=str(row.get("content") or ""), - layer=str(row.get("layer") or ""), - title=str(row.get("title") or ""), - metadata=dict(row.get("metadata", {}) or {}), - score=row.get("distance"), - location=location, - ) diff --git a/src/app/modules/agent/runtime/steps/explain/models.py b/src/app/modules/agent/runtime/steps/explain/models.py deleted file mode 100644 index 90552cd..0000000 --- a/src/app/modules/agent/runtime/steps/explain/models.py +++ /dev/null @@ -1,91 +0,0 @@ -from __future__ import annotations - -from typing import Any, Literal - -from pydantic import BaseModel, ConfigDict, Field - - -class ExplainHints(BaseModel): - model_config = ConfigDict(extra="forbid") - - paths: list[str] = Field(default_factory=list) - symbols: list[str] = Field(default_factory=list) - endpoints: list[str] = Field(default_factory=list) - commands: list[str] = Field(default_factory=list) - - -class ExplainIntent(BaseModel): - model_config = ConfigDict(extra="forbid") - - raw_query: str - normalized_query: str - keywords: list[str] = Field(default_factory=list) - hints: ExplainHints = Field(default_factory=ExplainHints) - include_tests: bool = False - expected_entry_types: list[Literal["http", "cli"]] = Field(default_factory=list) - depth: Literal["high", "medium", "deep"] = "medium" - - -class CodeLocation(BaseModel): - model_config = ConfigDict(extra="forbid") - - path: str - start_line: int | None = None - end_line: int | None = None - - -class LayeredRetrievalItem(BaseModel): - model_config = ConfigDict(extra="forbid") - - source: str - content: str - layer: str - title: str - metadata: dict[str, Any] = Field(default_factory=dict) - score: float | None = None - location: CodeLocation | None = None - - -class TracePath(BaseModel): - model_config = ConfigDict(extra="forbid") - - symbol_ids: list[str] = Field(default_factory=list) - score: float = 0.0 - entrypoint_id: str | None = None - notes: list[str] = Field(default_factory=list) - - -class EvidenceItem(BaseModel): - model_config = ConfigDict(extra="forbid") - - evidence_id: str - kind: Literal["entrypoint", "symbol", "edge", "excerpt"] - summary: str - location: CodeLocation | None = None - supports: list[str] = Field(default_factory=list) - - -class CodeExcerpt(BaseModel): - model_config = ConfigDict(extra="forbid") - - evidence_id: str - symbol_id: str | None = None - title: str - path: str - start_line: int | None = None - end_line: int | None = None - content: str - focus: str = "overview" - - -class ExplainPack(BaseModel): - model_config = ConfigDict(extra="forbid") - - intent: ExplainIntent - selected_entrypoints: list[LayeredRetrievalItem] = Field(default_factory=list) - seed_symbols: list[LayeredRetrievalItem] = Field(default_factory=list) - trace_paths: list[TracePath] = Field(default_factory=list) - evidence_index: dict[str, EvidenceItem] = Field(default_factory=dict) - code_excerpts: list[CodeExcerpt] = Field(default_factory=list) - missing: list[str] = Field(default_factory=list) - conflicts: list[str] = Field(default_factory=list) diff --git a/src/app/modules/agent/runtime/steps/explain/retriever_v2.py b/src/app/modules/agent/runtime/steps/explain/retriever_v2.py deleted file mode 100644 index bba502c..0000000 --- a/src/app/modules/agent/runtime/steps/explain/retriever_v2.py +++ /dev/null @@ -1,328 +0,0 @@ -from __future__ import annotations - -import logging -from typing import TYPE_CHECKING - -from app.modules.rag.contracts.enums import RagLayer -from app.modules.agent.runtime.steps.explain.intent_builder import ExplainIntentBuilder -from app.modules.agent.runtime.steps.explain.layered_gateway import LayerRetrievalResult, LayeredRetrievalGateway -from app.modules.agent.runtime.steps.explain.models import CodeExcerpt, EvidenceItem, ExplainPack, LayeredRetrievalItem -from app.modules.agent.runtime.steps.explain.source_excerpt_fetcher import SourceExcerptFetcher -from app.modules.agent.runtime.steps.explain.trace_builder import TraceBuilder -from app.modules.rag.retrieval.test_filter import exclude_tests_default, is_test_path - -LOGGER = logging.getLogger(__name__) -_MIN_EXCERPTS = 2 - -if TYPE_CHECKING: - from app.modules.agent.runtime.steps.explain.graph_repository import CodeGraphRepository - from app.modules.agent.runtime.steps.explain.models import ExplainIntent - - -class CodeExplainRetrieverV2: - def __init__( - self, - gateway: LayeredRetrievalGateway, - graph_repository: CodeGraphRepository, - intent_builder: ExplainIntentBuilder | None = None, - trace_builder: TraceBuilder | None = None, - excerpt_fetcher: SourceExcerptFetcher | None = None, - ) -> None: - self._gateway = gateway - self._graph = graph_repository - self._intent_builder = intent_builder or ExplainIntentBuilder() - self._trace_builder = trace_builder or TraceBuilder(graph_repository) - self._excerpt_fetcher = excerpt_fetcher or SourceExcerptFetcher(graph_repository) - - def build_pack( - self, - rag_session_id: str, - user_query: str, - *, - file_candidates: list[dict] | None = None, - ) -> ExplainPack: - intent = self._intent_builder.build(user_query) - path_prefixes = _path_prefixes(intent, file_candidates or []) - exclude_tests = exclude_tests_default() and not intent.include_tests - pack = self._run_pass(rag_session_id, intent, path_prefixes, exclude_tests=exclude_tests) - if exclude_tests and len(pack.code_excerpts) < _MIN_EXCERPTS: - self._merge_test_fallback(pack, rag_session_id, intent, path_prefixes) - self._log_pack(rag_session_id, pack) - return pack - - def _run_pass( - self, - rag_session_id: str, - intent: ExplainIntent, - path_prefixes: list[str], - *, - exclude_tests: bool, - ) -> ExplainPack: - missing: list[str] = [] - entrypoints_result = self._entrypoints(rag_session_id, intent, path_prefixes, exclude_tests=exclude_tests) - missing.extend(entrypoints_result.missing) - selected_entrypoints = self._filter_entrypoints(intent, entrypoints_result.items) - if not selected_entrypoints: - missing.append("layer:C3 empty") - seed_result = self._seed_symbols(rag_session_id, intent, path_prefixes, selected_entrypoints, exclude_tests=exclude_tests) - missing.extend(seed_result.missing) - seed_symbols = seed_result.items - if not seed_symbols: - missing.append("layer:C1 empty") - depth = 4 if intent.depth == "deep" else 3 if intent.depth == "medium" else 2 - trace_paths = self._trace_builder.build_paths(rag_session_id, seed_symbols, max_depth=depth) if seed_symbols else [] - excerpts, excerpt_evidence = self._excerpt_fetcher.fetch(rag_session_id, trace_paths) if trace_paths else ([], {}) - if not excerpts: - lexical_result = self._gateway.retrieve_lexical_code( - rag_session_id, - intent.normalized_query, - limit=6, - path_prefixes=path_prefixes or None, - exclude_tests=exclude_tests, - include_spans=True, - ) - missing.extend(lexical_result.missing) - excerpts, excerpt_evidence = _lexical_excerpts(lexical_result.items) - if not excerpts: - missing.append("layer:C0 empty") - evidence_index = _evidence_index(selected_entrypoints, seed_symbols) - evidence_index.update(excerpt_evidence) - missing.extend(_missing(selected_entrypoints, seed_symbols, trace_paths, excerpts)) - return ExplainPack( - intent=intent, - selected_entrypoints=selected_entrypoints, - seed_symbols=seed_symbols, - trace_paths=trace_paths, - evidence_index=evidence_index, - code_excerpts=excerpts, - missing=_cleanup_missing(_dedupe(missing), has_excerpts=bool(excerpts)), - conflicts=[], - ) - - def _merge_test_fallback( - self, - pack: ExplainPack, - rag_session_id: str, - intent: ExplainIntent, - path_prefixes: list[str], - ) -> None: - lexical_result = self._gateway.retrieve_lexical_code( - rag_session_id, - intent.normalized_query, - limit=6, - path_prefixes=path_prefixes or None, - exclude_tests=False, - include_spans=True, - ) - excerpt_offset = len([key for key in pack.evidence_index if key.startswith("excerpt_")]) - excerpts, evidence = _lexical_excerpts( - lexical_result.items, - start_index=excerpt_offset, - is_test_fallback=True, - ) - if not excerpts: - pack.missing = _dedupe(pack.missing + lexical_result.missing) - return - seen = {(item.path, item.start_line, item.end_line, item.content) for item in pack.code_excerpts} - for excerpt in excerpts: - key = (excerpt.path, excerpt.start_line, excerpt.end_line, excerpt.content) - if key in seen: - continue - pack.code_excerpts.append(excerpt) - seen.add(key) - pack.evidence_index.update(evidence) - pack.missing = _cleanup_missing(_dedupe(pack.missing + lexical_result.missing), has_excerpts=bool(pack.code_excerpts)) - - def _entrypoints( - self, - rag_session_id: str, - intent: ExplainIntent, - path_prefixes: list[str], - *, - exclude_tests: bool, - ) -> LayerRetrievalResult: - return self._gateway.retrieve_layer( - rag_session_id, - intent.normalized_query, - RagLayer.CODE_ENTRYPOINTS, - limit=6, - path_prefixes=path_prefixes or None, - exclude_tests=exclude_tests, - prefer_non_tests=True, - include_spans=True, - ) - - def _filter_entrypoints(self, intent: ExplainIntent, items: list[LayeredRetrievalItem]) -> list[LayeredRetrievalItem]: - if not intent.expected_entry_types: - return items[:3] - filtered = [item for item in items if str(item.metadata.get("entry_type") or "") in intent.expected_entry_types] - return filtered[:3] or items[:3] - - def _seed_symbols( - self, - rag_session_id: str, - intent: ExplainIntent, - path_prefixes: list[str], - entrypoints: list[LayeredRetrievalItem], - *, - exclude_tests: bool, - ) -> LayerRetrievalResult: - symbol_result = self._gateway.retrieve_layer( - rag_session_id, - intent.normalized_query, - RagLayer.CODE_SYMBOL_CATALOG, - limit=12, - path_prefixes=path_prefixes or None, - exclude_tests=exclude_tests, - prefer_non_tests=True, - include_spans=True, - ) - handlers: list[LayeredRetrievalItem] = [] - handler_ids = [str(item.metadata.get("handler_symbol_id") or "") for item in entrypoints] - if handler_ids: - handlers = self._graph.get_symbols_by_ids(rag_session_id, [item for item in handler_ids if item]) - seeds: list[LayeredRetrievalItem] = [] - seen: set[str] = set() - for item in handlers + symbol_result.items: - symbol_id = str(item.metadata.get("symbol_id") or "") - if not symbol_id or symbol_id in seen: - continue - seen.add(symbol_id) - seeds.append(item) - if len(seeds) >= 8: - break - return LayerRetrievalResult(items=seeds, missing=list(symbol_result.missing)) - - def _log_pack(self, rag_session_id: str, pack: ExplainPack) -> None: - prod_excerpt_count = len([excerpt for excerpt in pack.code_excerpts if not _is_test_excerpt(excerpt)]) - test_excerpt_count = len(pack.code_excerpts) - prod_excerpt_count - LOGGER.warning( - "code explain pack: rag_session_id=%s entrypoints=%s seeds=%s paths=%s excerpts=%s prod_excerpt_count=%s test_excerpt_count=%s missing=%s", - rag_session_id, - len(pack.selected_entrypoints), - len(pack.seed_symbols), - len(pack.trace_paths), - len(pack.code_excerpts), - prod_excerpt_count, - test_excerpt_count, - pack.missing, - ) - - -def _evidence_index( - entrypoints: list[LayeredRetrievalItem], - seed_symbols: list[LayeredRetrievalItem], -) -> dict[str, EvidenceItem]: - result: dict[str, EvidenceItem] = {} - for index, item in enumerate(entrypoints, start=1): - evidence_id = f"entrypoint_{index}" - result[evidence_id] = EvidenceItem( - evidence_id=evidence_id, - kind="entrypoint", - summary=item.title, - location=item.location, - supports=[str(item.metadata.get("handler_symbol_id") or "")], - ) - for index, item in enumerate(seed_symbols, start=1): - evidence_id = f"symbol_{index}" - result[evidence_id] = EvidenceItem( - evidence_id=evidence_id, - kind="symbol", - summary=item.title, - location=item.location, - supports=[str(item.metadata.get("symbol_id") or "")], - ) - return result - - -def _missing( - entrypoints: list[LayeredRetrievalItem], - seed_symbols: list[LayeredRetrievalItem], - trace_paths, - excerpts, -) -> list[str]: - missing: list[str] = [] - if not entrypoints: - missing.append("entrypoints") - if not seed_symbols: - missing.append("seed_symbols") - if not trace_paths: - missing.append("trace_paths") - if not excerpts: - missing.append("code_excerpts") - return missing - - -def _lexical_excerpts( - items: list[LayeredRetrievalItem], - *, - start_index: int = 0, - is_test_fallback: bool = False, -) -> tuple[list[CodeExcerpt], dict[str, EvidenceItem]]: - excerpts: list[CodeExcerpt] = [] - evidence_index: dict[str, EvidenceItem] = {} - for item in items: - evidence_id = f"excerpt_{start_index + len(evidence_index) + 1}" - location = item.location - evidence_index[evidence_id] = EvidenceItem( - evidence_id=evidence_id, - kind="excerpt", - summary=item.title or item.source, - location=location, - supports=[], - ) - focus = "lexical" - if _item_is_test(item): - focus = "test:lexical" - elif is_test_fallback: - focus = "lexical" - excerpts.append( - CodeExcerpt( - evidence_id=evidence_id, - symbol_id=str(item.metadata.get("symbol_id") or "") or None, - title=item.title or item.source, - path=item.source, - start_line=location.start_line if location else None, - end_line=location.end_line if location else None, - content=item.content, - focus=focus, - ) - ) - return excerpts, evidence_index - - -def _item_is_test(item: LayeredRetrievalItem) -> bool: - return bool(item.metadata.get("is_test")) or is_test_path(item.source) - - -def _is_test_excerpt(excerpt: CodeExcerpt) -> bool: - return excerpt.focus.startswith("test:") or is_test_path(excerpt.path) - - -def _path_prefixes(intent: ExplainIntent, file_candidates: list[dict]) -> list[str]: - values: list[str] = [] - for path in intent.hints.paths: - prefix = path.rsplit("/", 1)[0] if "/" in path else path - if prefix and prefix not in values: - values.append(prefix) - for item in file_candidates[:6]: - path = str(item.get("path") or "") - prefix = path.rsplit("/", 1)[0] if "/" in path else "" - if prefix and prefix not in values: - values.append(prefix) - return values - - -def _cleanup_missing(values: list[str], *, has_excerpts: bool) -> list[str]: - if not has_excerpts: - return values - return [value for value in values if value not in {"code_excerpts", "layer:C0 empty"}] - - -def _dedupe(values: list[str]) -> list[str]: - result: list[str] = [] - for value in values: - item = value.strip() - if item and item not in result: - result.append(item) - return result diff --git a/src/app/modules/agent/runtime/steps/explain/source_excerpt_fetcher.py b/src/app/modules/agent/runtime/steps/explain/source_excerpt_fetcher.py deleted file mode 100644 index cd3c4ae..0000000 --- a/src/app/modules/agent/runtime/steps/explain/source_excerpt_fetcher.py +++ /dev/null @@ -1,53 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING - -from app.modules.agent.runtime.steps.explain.excerpt_planner import ExcerptPlanner -from app.modules.agent.runtime.steps.explain.models import CodeExcerpt, EvidenceItem, TracePath -from app.modules.rag.retrieval.test_filter import is_test_path - -if TYPE_CHECKING: - from app.modules.agent.runtime.steps.explain.graph_repository import CodeGraphRepository - - -class SourceExcerptFetcher: - def __init__(self, graph_repository: CodeGraphRepository, planner: ExcerptPlanner | None = None) -> None: - self._graph = graph_repository - self._planner = planner or ExcerptPlanner() - - def fetch( - self, - rag_session_id: str, - trace_paths: list[TracePath], - *, - max_excerpts: int = 40, - ) -> tuple[list[CodeExcerpt], dict[str, EvidenceItem]]: - ordered_symbol_ids: list[str] = [] - for path in trace_paths: - for symbol_id in path.symbol_ids: - if symbol_id and symbol_id not in ordered_symbol_ids: - ordered_symbol_ids.append(symbol_id) - chunks = self._graph.get_chunks_by_symbol_ids(rag_session_id, ordered_symbol_ids) - excerpts: list[CodeExcerpt] = [] - evidence_index: dict[str, EvidenceItem] = {} - for chunk in chunks: - symbol_id = str(chunk.metadata.get("symbol_id") or "") - evidence_id = f"excerpt_{len(evidence_index) + 1}" - location = chunk.location - evidence_index[evidence_id] = EvidenceItem( - evidence_id=evidence_id, - kind="excerpt", - summary=chunk.title, - location=location, - supports=[symbol_id] if symbol_id else [], - ) - is_test_chunk = bool(chunk.metadata.get("is_test")) or is_test_path(location.path if location else chunk.source) - for excerpt in self._planner.plan(chunk, evidence_id=evidence_id, symbol_id=symbol_id): - if len(excerpts) >= max_excerpts: - break - if is_test_chunk and not excerpt.focus.startswith("test:"): - excerpt.focus = f"test:{excerpt.focus}" - excerpts.append(excerpt) - if len(excerpts) >= max_excerpts: - break - return excerpts, evidence_index diff --git a/src/app/modules/agent/runtime/steps/explain/trace_builder.py b/src/app/modules/agent/runtime/steps/explain/trace_builder.py deleted file mode 100644 index a9c2826..0000000 --- a/src/app/modules/agent/runtime/steps/explain/trace_builder.py +++ /dev/null @@ -1,102 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING - -from app.modules.agent.runtime.steps.explain.models import LayeredRetrievalItem, TracePath - -if TYPE_CHECKING: - from app.modules.agent.runtime.steps.explain.graph_repository import CodeGraphRepository - - -class TraceBuilder: - def __init__(self, graph_repository: CodeGraphRepository) -> None: - self._graph = graph_repository - - def build_paths( - self, - rag_session_id: str, - seed_symbols: list[LayeredRetrievalItem], - *, - max_depth: int, - max_paths: int = 3, - edge_types: list[str] | None = None, - ) -> list[TracePath]: - edges_filter = edge_types or ["calls", "imports", "inherits", "instantiates", "reads_attr", "writes_attr"] - symbol_map = self._symbol_map(seed_symbols) - paths: list[TracePath] = [] - for seed in seed_symbols: - seed_id = str(seed.metadata.get("symbol_id") or "") - if not seed_id: - continue - queue: list[tuple[list[str], float, list[str]]] = [([seed_id], 0.0, [])] - while queue and len(paths) < max_paths * 3: - current_path, score, notes = queue.pop(0) - src_symbol_id = current_path[-1] - out_edges = self._graph.get_out_edges(rag_session_id, [src_symbol_id], edges_filter, limit_per_src=4) - if not out_edges or len(current_path) >= max_depth: - paths.append(TracePath(symbol_ids=current_path, score=score, notes=notes)) - continue - for edge in out_edges: - metadata = edge.metadata - dst_symbol_id = str(metadata.get("dst_symbol_id") or "") - next_notes = list(notes) - next_score = score + self._edge_score(edge, symbol_map.get(src_symbol_id)) - if not dst_symbol_id: - dst_ref = str(metadata.get("dst_ref") or "") - package_hint = self._package_hint(symbol_map.get(src_symbol_id)) - resolved = self._graph.resolve_symbol_by_ref(rag_session_id, dst_ref, package_hint=package_hint) - if resolved is not None: - dst_symbol_id = str(resolved.metadata.get("symbol_id") or "") - symbol_map[dst_symbol_id] = resolved - next_score += 2.0 - next_notes.append(f"resolved:{dst_ref}") - if not dst_symbol_id or dst_symbol_id in current_path: - paths.append(TracePath(symbol_ids=current_path, score=next_score, notes=next_notes)) - continue - if dst_symbol_id not in symbol_map: - symbols = self._graph.get_symbols_by_ids(rag_session_id, [dst_symbol_id]) - if symbols: - symbol_map[dst_symbol_id] = symbols[0] - queue.append((current_path + [dst_symbol_id], next_score, next_notes)) - unique = self._unique_paths(paths) - unique.sort(key=lambda item: item.score, reverse=True) - return unique[:max_paths] or [TracePath(symbol_ids=[seed.metadata.get("symbol_id", "")], score=0.0) for seed in seed_symbols[:1]] - - def _edge_score(self, edge: LayeredRetrievalItem, source_symbol: LayeredRetrievalItem | None) -> float: - metadata = edge.metadata - score = 1.0 - if str(metadata.get("resolution") or "") == "resolved": - score += 2.0 - source_path = source_symbol.source if source_symbol is not None else "" - if source_path and edge.source == source_path: - score += 1.0 - if "tests/" in edge.source or "/tests/" in edge.source: - score -= 3.0 - return score - - def _package_hint(self, symbol: LayeredRetrievalItem | None) -> str | None: - if symbol is None: - return None - package = str(symbol.metadata.get("package_or_module") or "") - if not package: - return None - return ".".join(package.split(".")[:-1]) or package - - def _symbol_map(self, items: list[LayeredRetrievalItem]) -> dict[str, LayeredRetrievalItem]: - result: dict[str, LayeredRetrievalItem] = {} - for item in items: - symbol_id = str(item.metadata.get("symbol_id") or "") - if symbol_id: - result[symbol_id] = item - return result - - def _unique_paths(self, items: list[TracePath]) -> list[TracePath]: - result: list[TracePath] = [] - seen: set[tuple[str, ...]] = set() - for item in items: - key = tuple(symbol_id for symbol_id in item.symbol_ids if symbol_id) - if not key or key in seen: - continue - seen.add(key) - result.append(item) - return result diff --git a/src/app/modules/agent/runtime/steps/finalization/__init__.py b/src/app/modules/agent/runtime/steps/finalization/__init__.py deleted file mode 100644 index e59188d..0000000 --- a/src/app/modules/agent/runtime/steps/finalization/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -"""Финальная сборка: repair черновика и сборка RuntimeFinalResult.""" - -from app.modules.agent.runtime.steps.finalization.repair import RuntimeAnswerRepairService -from app.modules.agent.runtime.steps.finalization.result_assembler import assemble_final_result - -__all__ = ["RuntimeAnswerRepairService", "assemble_final_result"] diff --git a/src/app/modules/agent/runtime/steps/finalization/repair.py b/src/app/modules/agent/runtime/steps/finalization/repair.py deleted file mode 100644 index f57eb6c..0000000 --- a/src/app/modules/agent/runtime/steps/finalization/repair.py +++ /dev/null @@ -1,68 +0,0 @@ -"""Сервис починки черновика ответа по результатам post-evidence gate (LLM repair).""" - -from __future__ import annotations - -import json - -from app.modules.agent.runtime.steps.gates.post.post_gate import RuntimeValidationResult -from app.modules.agent.llm import AgentLlmService - - -class RuntimeAnswerRepairService: - def __init__(self, llm: AgentLlmService) -> None: - self._llm = llm - - def repair( - self, - *, - draft_answer: str, - validation: RuntimeValidationResult, - prompt_payload: str, - ) -> str: - repair_focus = self._repair_focus(validation.reasons) - repair_input = json.dumps( - { - "draft_answer": draft_answer, - "validation_reasons": validation.reasons, - "repair_focus": repair_focus, - "prompt_payload": prompt_payload, - }, - ensure_ascii=False, - indent=2, - ) - return self._llm.generate( - "code_qa_repair_answer", - repair_input, - log_context="graph.project_qa.code_qa.repair", - ).strip() - - def _repair_focus(self, reasons: list[str]) -> list[str]: - mapping = { - "missing_concrete_methods": "missing_concrete_methods", - "missing_concrete_calls": "missing_concrete_calls", - "missing_concrete_dependencies": "missing_concrete_dependencies", - "missing_concrete_fields": "missing_concrete_fields", - "ignores_concrete_explain_facts": "too_vague_for_explain", - "too_vague_for_explain": "too_vague_for_explain", - "missing_concrete_components": "missing_concrete_components", - "missing_concrete_relations": "missing_concrete_relations", - "missing_relation_verbs": "missing_relation_verbs", - "target_mentioned_but_no_relations": "missing_concrete_relations", - "too_vague_for_architecture": "too_vague_for_architecture", - "missing_flow_steps": "missing_flow_steps", - "missing_sequence_edges": "missing_sequence_edges", - "missing_ordered_flow_steps": "missing_flow_steps", - "too_vague_for_trace_flow": "too_vague_for_trace_flow", - "semantic_labels_without_code_edges": "semantic_labels_without_code_edges", - "contains_retrieval_artifacts": "contains_retrieval_artifacts", - "methods_as_primary_components": "methods_as_primary_components", - "overclaims_trace_completeness": "overclaims_trace_completeness", - } - seen: set[str] = set() - result: list[str] = [] - for reason in reasons: - focus = mapping.get(reason) - if focus and focus not in seen: - seen.add(focus) - result.append(focus) - return result or ["tighten_to_evidence"] diff --git a/src/app/modules/agent/runtime/steps/finalization/result_assembler.py b/src/app/modules/agent/runtime/steps/finalization/result_assembler.py deleted file mode 100644 index 23c9b9e..0000000 --- a/src/app/modules/agent/runtime/steps/finalization/result_assembler.py +++ /dev/null @@ -1,79 +0,0 @@ -"""Сборка финального результата пайплайна: диагностика и RuntimeFinalResult.""" - -from __future__ import annotations - -import logging -from typing import Any - -from app.modules.agent.runtime.steps.context.diagnostics import build_diagnostics_report -from app.modules.agent.runtime.steps.gates.post.post_gate import RuntimePostEvidenceGate, RuntimeValidationResult -from app.modules.agent.runtime.models import RuntimeDraftAnswer, RuntimeExecutionState, RuntimeFinalResult - -LOGGER = logging.getLogger(__name__) - - -def assemble_final_result( - state: RuntimeExecutionState, - *, - draft: RuntimeDraftAnswer | None, - final_answer: str, - repair_used: bool, - llm_used: bool, - validation: RuntimeValidationResult | None = None, - timings_ms: dict[str, int] | None = None, - runtime_trace: list[dict] | None = None, - answer_policy_branch: str = "", - decision_reason: str = "", - pre_gate_input: dict[str, Any] | None = None, - post_gate_snapshot: dict[str, Any] | None = None, - resolved_target: str | None = None, - post_gate: RuntimePostEvidenceGate | None = None, -) -> RuntimeFinalResult: - diagnostics = build_diagnostics_report( - router_result=state.router_result, - retrieval_request=state.retrieval_request, - retrieval_result=state.retrieval_result, - evidence_bundle=state.evidence_pack, - answer_mode=state.answer_mode, - timings_ms=timings_ms or {}, - resolved_target=resolved_target, - answer_policy_branch=answer_policy_branch, - decision_reason=decision_reason, - evidence_gate_input=pre_gate_input or {}, - post_evidence_gate=post_gate_snapshot or {}, - ) - if validation is None and post_gate is not None and state.retrieval_request is not None: - validation = post_gate.validate( - answer=final_answer, - answer_mode=state.answer_mode, - degraded_message=state.degraded_message, - sub_intent=state.retrieval_request.sub_intent, - user_query=state.user_query, - evidence_pack=state.evidence_pack, - ) - elif validation is None: - validation = RuntimeValidationResult(passed=True, action="return") - - result = RuntimeFinalResult( - final_answer=final_answer.strip(), - answer_mode=state.answer_mode, - repair_used=repair_used, - llm_used=llm_used, - draft_answer=draft, - validation=validation, - router_result=state.router_result, - retrieval_request=state.retrieval_request, - retrieval_result=state.retrieval_result, - evidence_pack=state.evidence_pack, - diagnostics=diagnostics, - runtime_trace=list(runtime_trace or []), - ) - LOGGER.warning( - "agent runtime executed: intent=%s sub_intent=%s answer_mode=%s repair_used=%s llm_used=%s", - state.router_result.intent if state.router_result else None, - state.router_result.query_plan.sub_intent if state.router_result and state.router_result.query_plan else None, - result.answer_mode, - result.repair_used, - result.llm_used, - ) - return result diff --git a/src/app/modules/agent/runtime/steps/gates/__init__.py b/src/app/modules/agent/runtime/steps/gates/__init__.py deleted file mode 100644 index c7e0e58..0000000 --- a/src/app/modules/agent/runtime/steps/gates/__init__.py +++ /dev/null @@ -1,11 +0,0 @@ -"""Pre- и post-evidence gates пайплайна.""" - -from app.modules.agent.runtime.steps.gates.pre.evidence_gate import EvidenceGateDecision, evaluate_evidence -from app.modules.agent.runtime.steps.gates.post.post_gate import RuntimePostEvidenceGate, RuntimeValidationResult - -__all__ = [ - "EvidenceGateDecision", - "evaluate_evidence", - "RuntimePostEvidenceGate", - "RuntimeValidationResult", -] diff --git a/src/app/modules/agent/runtime/steps/gates/post/__init__.py b/src/app/modules/agent/runtime/steps/gates/post/__init__.py deleted file mode 100644 index c7cd976..0000000 --- a/src/app/modules/agent/runtime/steps/gates/post/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Post-evidence gate: валидация черновика ответа и решение repair/return.""" - -from app.modules.agent.runtime.steps.gates.post.post_gate import RuntimePostEvidenceGate, RuntimeValidationResult - -__all__ = ["RuntimePostEvidenceGate", "RuntimeValidationResult"] diff --git a/src/app/modules/agent/runtime/steps/gates/post/post_gate.py b/src/app/modules/agent/runtime/steps/gates/post/post_gate.py deleted file mode 100644 index 44e0d60..0000000 --- a/src/app/modules/agent/runtime/steps/gates/post/post_gate.py +++ /dev/null @@ -1,287 +0,0 @@ -"""Post-evidence gate: валидация черновика ответа и решение repair/return.""" - -from __future__ import annotations - -import re - -from pydantic import BaseModel, ConfigDict, Field - -from app.modules.agent.runtime.steps.context.answer_fact_curator import build_curated_answer_facts -from app.modules.agent.runtime.steps.context.contracts import EvidenceBundle - - -class RuntimeValidationResult(BaseModel): - model_config = ConfigDict(extra="forbid") - - passed: bool = False - action: str = "return" - reasons: list[str] = Field(default_factory=list) - - -_TOKEN_RE = re.compile(r"[a-zA-Zа-яА-Я0-9_/]+") -_VAGUE_PHRASES = ( - "ряд аргументов", - "имеет responsibilities", - "имеет responsibility", - "ключевой компонент", - "центральный компонент", - "играет роль", - "управляет системой", - "этап пайплайна", - "инициализация сервисов", - "регистрация основных служб", - "различные аргументы", - "различные подпакеты", - "основные службы", - "представляет собой", - "используется в службах", -) -_OPTIMISTIC_TRACE_CLAIMS = ( - "полностью восстанавливается", - "полный поток выполнения", - "полностью прослеживается", - "полный поток виден", - "полная цепочка", - "весь поток", -) - - -class RuntimePostEvidenceGate: - def validate( - self, - *, - answer: str, - answer_mode: str, - degraded_message: str, - sub_intent: str, - user_query: str, - evidence_pack: EvidenceBundle | None, - ) -> RuntimeValidationResult: - normalized = (answer or "").strip() - if not normalized: - return RuntimeValidationResult(passed=False, action="repair", reasons=["empty_answer"]) - if answer_mode in {"degraded", "insufficient"} and "недостат" not in normalized.lower(): - return RuntimeValidationResult(passed=False, action="repair", reasons=["degraded_answer_missing_guardrail"]) - if answer_mode == "not_found" and "не найден" not in normalized.lower(): - return RuntimeValidationResult(passed=False, action="repair", reasons=["not_found_answer_missing_phrase"]) - if answer_mode == "ambiguous" and "не удалось однозначно разрешить" not in normalized.lower(): - return RuntimeValidationResult(passed=False, action="repair", reasons=["ambiguous_answer_missing_phrase"]) - if degraded_message and answer_mode != "normal" and len(normalized) < 24: - return RuntimeValidationResult(passed=False, action="repair", reasons=["answer_too_short"]) - if answer_mode != "normal" or evidence_pack is None: - return RuntimeValidationResult(passed=True, action="return") - - reasons = self._normal_answer_reasons(normalized.lower(), sub_intent.upper(), user_query, evidence_pack) - if reasons: - return RuntimeValidationResult(passed=False, action="repair", reasons=_dedupe(reasons)) - return RuntimeValidationResult(passed=True, action="return") - - def _normal_answer_reasons(self, answer: str, sub_intent: str, user_query: str, evidence_pack: EvidenceBundle) -> list[str]: - reasons: list[str] = [] - if sub_intent == "FIND_ENTRYPOINTS": - reasons.extend(self._validate_entrypoints(answer, user_query, evidence_pack)) - elif sub_intent == "EXPLAIN": - reasons.extend(self._validate_explain(answer, evidence_pack)) - elif sub_intent == "ARCHITECTURE": - reasons.extend(self._validate_architecture(answer, evidence_pack)) - elif sub_intent == "TRACE_FLOW": - reasons.extend(self._validate_trace_flow(answer, evidence_pack)) - return reasons - - def _validate_entrypoints(self, answer: str, user_query: str, evidence_pack: EvidenceBundle) -> list[str]: - confirmed = [item for item in list(evidence_pack.entrypoints or []) if item.get("http_method") and item.get("route_path")] - if not confirmed: - return [] - query_tokens = self._tokens(user_query) - matched = [item for item in confirmed if self._tokens(str(item.get("route_path") or "")) & query_tokens] - required = matched or confirmed[:1] - reasons = [] - if any(f"{item['http_method']} {item['route_path']}".lower() not in answer for item in required): - reasons.append("missing_confirmed_route") - if matched and any(token in query_tokens for token in {"health", "/health"}) and "не обнаруж" in answer: - reasons.append("contradicts_confirmed_route") - return reasons - - def _validate_explain(self, answer: str, evidence_pack: EvidenceBundle) -> list[str]: - facts = build_curated_answer_facts(evidence_pack) - explain = dict(facts.get("explain") or {}) - reasons = self._validate_target_focus(answer, evidence_pack) - reasons.extend(self._vagueness_reasons(answer, "explain")) - - methods = list(explain.get("required_methods") or []) - calls = list(explain.get("required_calls") or []) - dependencies = list(explain.get("required_dependencies") or []) - fields = list(explain.get("required_fields") or []) - has_any_required = bool(methods or calls or dependencies or fields) - matches = 0 - if methods and not self._mentions_fact_group(answer, methods): - reasons.append("missing_concrete_methods") - elif methods: - matches += 1 - if calls and not self._mentions_fact_group(answer, calls): - reasons.append("missing_concrete_calls") - elif calls: - matches += 1 - if dependencies and not self._mentions_fact_group(answer, dependencies): - reasons.append("missing_concrete_dependencies") - elif dependencies: - matches += 1 - if fields and not self._mentions_fact_group(answer, fields): - reasons.append("missing_concrete_fields") - elif fields: - matches += 1 - if has_any_required and matches == 0: - reasons.append("too_vague_for_explain") - if has_any_required and self._answer_is_generic_with_facts(answer, methods + calls + dependencies + fields): - reasons.append("ignores_concrete_explain_facts") - if self._semantic_leakage(answer, facts, has_concrete_support=matches > 0): - reasons.append("semantic_labels_without_code_edges") - return reasons - - def _validate_architecture(self, answer: str, evidence_pack: EvidenceBundle) -> list[str]: - facts = build_curated_answer_facts(evidence_pack) - architecture = dict(facts.get("architecture") or {}) - reasons = self._validate_target_focus(answer, evidence_pack) - reasons.extend(self._vagueness_reasons(answer, "architecture")) - - components = list(architecture.get("required_components") or []) - relations = list(architecture.get("required_relations") or []) - verbs = list(architecture.get("required_relation_verbs") or []) - if components and not self._mentions_fact_group(answer, components): - reasons.append("missing_concrete_components") - if relations and not self._mentions_relations(answer, relations): - reasons.append("missing_concrete_relations") - if verbs and not self._mentions_fact_group(answer, verbs): - reasons.append("missing_relation_verbs") - forbidden = architecture.get("forbidden_labels") or [] - if any(label.lower() in answer.lower() for label in forbidden): - reasons.append("contains_retrieval_artifacts") - if self._methods_dominate_components(answer, components): - reasons.append("methods_as_primary_components") - has_relations = self._mentions_relations(answer, relations) - has_verbs = self._mentions_fact_group(answer, verbs) - if relations and (not has_relations or not has_verbs): - reasons.append("too_vague_for_architecture") - target = str(evidence_pack.resolved_target or "").strip().lower() - if target and target in answer and relations and not has_relations: - reasons.append("target_mentioned_but_no_relations") - if self._semantic_leakage(answer, facts, has_concrete_support=has_relations): - reasons.append("semantic_labels_without_code_edges") - return reasons - - def _validate_trace_flow(self, answer: str, evidence_pack: EvidenceBundle) -> list[str]: - facts = build_curated_answer_facts(evidence_pack) - trace = dict(facts.get("trace_flow") or {}) - reasons = self._validate_target_focus(answer, evidence_pack) - reasons.extend(self._vagueness_reasons(answer, "trace_flow")) - - steps = list(trace.get("required_flow_steps") or []) - calls = list(trace.get("required_calls") or []) - ordered_descriptions = list(trace.get("ordered_step_descriptions") or []) - if steps and not self._mentions_steps(answer, steps): - reasons.append("missing_flow_steps") - if steps and not self._mentions_relations(answer, steps): - reasons.append("missing_sequence_edges") - if calls and not self._mentions_fact_group(answer, calls): - reasons.append("missing_concrete_calls") - if any(claim.lower() in answer.lower() for claim in _OPTIMISTIC_TRACE_CLAIMS): - reasons.append("overclaims_trace_completeness") - if steps and not (self._mentions_steps(answer, steps) and self._mentions_relations(answer, steps)): - reasons.append("too_vague_for_trace_flow") - if ordered_descriptions and not self._mentions_ordered_steps(answer, ordered_descriptions): - reasons.append("missing_ordered_flow_steps") - return reasons - - def _validate_target_focus(self, answer: str, evidence_pack: EvidenceBundle) -> list[str]: - target = str(evidence_pack.resolved_target or "").strip().lower() - return [] if not target or target in answer else ["missing_resolved_target"] - - def _mentions_fact_group(self, answer: str, values: list[str]) -> bool: - return any(alias in answer for value in values for alias in _aliases(value)) - - def _mentions_relations(self, answer: str, relations: list[dict]) -> bool: - for relation in relations: - source = str(relation.get("source") or "").lower() - target = str(relation.get("target") or "").lower() - verb = str(relation.get("verb") or "").lower() - if source and target and source in answer and target in answer: - return True - if source and verb and target and source in answer and verb in answer and target in answer: - return True - return False - - def _mentions_steps(self, answer: str, steps: list[dict]) -> bool: - if "сначала" in answer and "затем" in answer: - return True - numeric_steps = sum(1 for marker in ("1.", "2.", "3.") if marker in answer) - if numeric_steps >= 2: - return True - mentioned = sum(1 for step in steps[:3] if self._mentions_relations(answer, [step])) - return mentioned >= min(2, len(steps[:3])) - - def _mentions_ordered_steps(self, answer: str, ordered_descriptions: list[str]) -> bool: - """Проверяет, что в ответе отражена хотя бы часть упорядоченных шагов (источник/цель/глагол).""" - if not ordered_descriptions: - return True - answer_lower = answer.lower() - hits = 0 - for desc in ordered_descriptions[:5]: - parts = desc.replace(".", " ").split() - if len(parts) >= 3 and any(p in answer_lower for p in parts if len(p) > 2): - hits += 1 - return hits >= min(2, len(ordered_descriptions)) - - def _answer_is_generic_with_facts( - self, answer: str, concrete_values: list[str] - ) -> bool: - """True, если в ответе есть общие фразы, но почти нет конкретных имён из списка.""" - if not concrete_values: - return False - mentioned = sum(1 for v in concrete_values if any(a in answer.lower() for a in _aliases(v))) - return mentioned == 0 and len(answer) > 100 - - def _methods_dominate_components(self, answer: str, components: list[str]) -> bool: - method_like = re.findall(r"\b[a-z_]+\(\)", answer) - component_hits = sum(1 for component in components if component.lower() in answer) - return bool(method_like) and component_hits == 0 - - def _semantic_leakage(self, answer: str, facts: dict, *, has_concrete_support: bool) -> bool: - if has_concrete_support: - return False - semantic_roles = [str(item.get("role") or "").strip().lower() for item in facts.get("semantic_hints") or [] if str(item.get("role") or "").strip()] - return bool(semantic_roles) and any(role in answer for role in semantic_roles) - - def _vagueness_reasons(self, answer: str, scenario: str) -> list[str]: - if any(phrase in answer for phrase in _VAGUE_PHRASES): - return [f"too_vague_for_{scenario}"] - return [] - - def _tokens(self, value: str) -> set[str]: - return {token.lower().strip("/") for token in _TOKEN_RE.findall(value or "") if token.strip("/")} - - -def _aliases(value: str) -> list[str]: - text = str(value or "").strip().lower() - if not text: - return [] - aliases = [text] - bare = text.removesuffix("()") - if bare != text: - aliases.append(bare) - tail = bare.rsplit(".", 1)[-1] - if tail and tail not in aliases: - aliases.append(tail) - if tail and f"{tail}()" not in aliases: - aliases.append(f"{tail}()") - return aliases - - -def _dedupe(values: list[str]) -> list[str]: - seen: set[str] = set() - result: list[str] = [] - for value in values: - if value in seen: - continue - seen.add(value) - result.append(value) - return result diff --git a/src/app/modules/agent/runtime/steps/gates/pre/__init__.py b/src/app/modules/agent/runtime/steps/gates/pre/__init__.py deleted file mode 100644 index 3881cc2..0000000 --- a/src/app/modules/agent/runtime/steps/gates/pre/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Pre-evidence gate: проверка достаточности evidence перед вызовом LLM.""" - -from app.modules.agent.runtime.steps.gates.pre.evidence_gate import EvidenceGateDecision, evaluate_evidence - -__all__ = ["EvidenceGateDecision", "evaluate_evidence"] diff --git a/src/app/modules/agent/runtime/steps/gates/pre/evidence_gate.py b/src/app/modules/agent/runtime/steps/gates/pre/evidence_gate.py deleted file mode 100644 index 633ce11..0000000 --- a/src/app/modules/agent/runtime/steps/gates/pre/evidence_gate.py +++ /dev/null @@ -1,72 +0,0 @@ -"""Проверка достаточности evidence для пайплайна CODE_QA.""" - -from __future__ import annotations - -from dataclasses import dataclass, field - -from app.modules.agent.runtime.steps.context.contracts import EvidenceBundle - - -@dataclass(slots=True) -class EvidenceGateDecision: - passed: bool - failure_reasons: list[str] = field(default_factory=list) - degraded_message: str = "" - - -def evaluate_evidence(bundle: EvidenceBundle) -> EvidenceGateDecision: - sub = (bundle.resolved_sub_intent or "EXPLAIN").upper() - reasons: list[str] = [] - - if sub == "OPEN_FILE": - if not bundle.resolved_target and not bundle.file_candidates: - reasons.append("path_scope_empty") - if bundle.evidence_count < 1: - reasons.append("layer_c0_empty") - elif sub == "EXPLAIN": - if bundle.target_type == "symbol" and not bundle.resolved_target: - reasons.append("target_not_resolved") - if bundle.evidence_count < 2: - reasons.append("insufficient_evidence") - if bundle.target_type == "symbol" and not bundle.relations and bundle.evidence_count < 2: - reasons.append("layer_c2_empty") - elif sub == "FIND_TESTS": - if not bundle.resolved_target and not bundle.target_symbol_candidates: - reasons.append("target_not_resolved") - if not bundle.test_evidence: - reasons.append("tests_not_found") - elif sub == "FIND_ENTRYPOINTS": - if not bundle.entrypoints: - reasons.append("entrypoints_not_found") - else: - if bundle.evidence_count < 1: - reasons.append("insufficient_evidence") - - passed = len(reasons) == 0 - if not passed: - bundle.failure_reasons = list(reasons) - bundle.sufficient = False - else: - bundle.sufficient = True - - degraded = _degraded_message(bundle, reasons) - return EvidenceGateDecision(passed=passed, failure_reasons=reasons, degraded_message=degraded) - - -def _degraded_message(bundle: EvidenceBundle, reasons: list[str]) -> str: - if not reasons: - return "" - lines = ["Insufficient evidence for a confident answer."] - if "target_not_resolved" in reasons: - lines.append("Target symbol or file could not be resolved.") - if "path_scope_empty" in reasons: - lines.append("No matching path or file found.") - if "layer_c0_empty" in reasons: - lines.append("No code chunks retrieved.") - if "tests_not_found" in reasons: - lines.append("No test candidates found.") - if "entrypoints_not_found" in reasons: - lines.append("No entrypoints found.") - if "insufficient_evidence" in reasons: - lines.append(f"Evidence count: {bundle.evidence_count} (minimum required not met).") - return " ".join(lines).strip() diff --git a/src/app/modules/agent/runtime/steps/generation/__init__.py b/src/app/modules/agent/runtime/steps/generation/__init__.py deleted file mode 100644 index f9c9eef..0000000 --- a/src/app/modules/agent/runtime/steps/generation/__init__.py +++ /dev/null @@ -1,11 +0,0 @@ -"""Выбор промпта, сборка payload и вызов LLM для генерации ответа.""" - -from app.modules.agent.runtime.steps.generation.prompt_selector import RuntimePromptSelector -from app.modules.agent.runtime.steps.generation.prompt_payload_builder import RuntimePromptPayloadBuilder -from app.modules.agent.runtime.steps.generation.generator import RuntimeAnswerGenerator - -__all__ = [ - "RuntimePromptSelector", - "RuntimePromptPayloadBuilder", - "RuntimeAnswerGenerator", -] diff --git a/src/app/modules/agent/runtime/steps/generation/generator.py b/src/app/modules/agent/runtime/steps/generation/generator.py deleted file mode 100644 index 52d5b62..0000000 --- a/src/app/modules/agent/runtime/steps/generation/generator.py +++ /dev/null @@ -1,18 +0,0 @@ -"""Тонкая обёртка над LLM для генерации ответа по имени промпта и payload.""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -if TYPE_CHECKING: - from app.modules.agent.llm import AgentLlmService - - -class RuntimeAnswerGenerator: - """Делегирует вызов LLM для генерации черновика ответа.""" - - def __init__(self, llm: AgentLlmService) -> None: - self._llm = llm - - def generate(self, prompt_name: str, payload: str, *, log_context: str = "graph.project_qa.code_qa.answer") -> str: - return self._llm.generate(prompt_name, payload, log_context=log_context).strip() diff --git a/src/app/modules/agent/runtime/steps/generation/prompt_payload_builder.py b/src/app/modules/agent/runtime/steps/generation/prompt_payload_builder.py deleted file mode 100644 index c84926e..0000000 --- a/src/app/modules/agent/runtime/steps/generation/prompt_payload_builder.py +++ /dev/null @@ -1,140 +0,0 @@ -"""Сборка JSON-полезной нагрузки для системного промпта по synthesis_input и evidence_pack.""" - -from __future__ import annotations - -import json -import re - -from app.modules.agent.runtime.steps.context.contracts import AnswerSynthesisInput, EvidenceBundle - -_LAYER_GUIDE = ( - "- C0_SOURCE_CHUNKS: фактический код, это основной источник деталей.\n" - "- C1_SYMBOL_CATALOG: объявления и сигнатуры символов.\n" - "- C2_DEPENDENCY_GRAPH: связи вызовов и зависимостей.\n" - "- C3_ENTRYPOINTS: подтвержденные точки входа.\n" - "- C4_SEMANTIC_ROLES: вспомогательная роль компонента, использовать осторожно." -) -_TOKEN_RE = re.compile(r"[a-zA-Zа-яА-Я0-9_/]+") - - -class RuntimePromptPayloadBuilder: - def build( - self, - *, - user_query: str, - synthesis_input: AnswerSynthesisInput, - evidence_pack: EvidenceBundle, - answer_mode: str, - ) -> str: - payload = { - "user_query": user_query, - "resolved_scenario": synthesis_input.resolved_scenario, - "resolved_target": synthesis_input.resolved_target, - "answer_mode": answer_mode, - "fast_context": synthesis_input.fast_context, - "deep_context": synthesis_input.deep_context, - "evidence_summary": synthesis_input.evidence_summary, - "semantic_hints": synthesis_input.semantic_hints, - "diagnostic_hints": synthesis_input.diagnostic_hints, - "retrieval_summary": evidence_pack.retrieval_summary, - "confirmed_entrypoints": self._entrypoints(user_query, evidence_pack), - "required_entrypoints": self._required_entrypoints(user_query, evidence_pack), - "layer_guide": _LAYER_GUIDE, - } - payload.update(self._scenario_payload(synthesis_input)) - return json.dumps(payload, ensure_ascii=False, indent=2) - - def _entrypoints(self, user_query: str, evidence_pack: EvidenceBundle) -> list[dict]: - query_tokens = self._tokens(user_query) - enriched: list[dict] = [] - for item in list(evidence_pack.entrypoints or []): - route_path = str(item.get("route_path") or "").strip() - http_method = str(item.get("http_method") or "").strip().upper() - if not http_method: - continue - candidate = { - "http_method": http_method, - "route_path": route_path, - "display": f"{http_method} {route_path}".strip(), - "handler_symbol": str(item.get("handler_symbol") or "").strip(), - "path": str(item.get("path") or "").strip(), - "start_line": item.get("start_line"), - "end_line": item.get("end_line"), - "query_match": self._matches_query(route_path, query_tokens), - } - if candidate not in enriched: - enriched.append(candidate) - return sorted(enriched, key=lambda item: (-int(bool(item["query_match"])), -int(bool(item["route_path"])), len(item["route_path"] or "/"), item["display"])) - - def _required_entrypoints(self, user_query: str, evidence_pack: EvidenceBundle) -> list[str]: - return [item["display"] for item in self._entrypoints(user_query, evidence_pack) if item["query_match"] and item["route_path"]] - - def _scenario_payload(self, synthesis_input: AnswerSynthesisInput) -> dict: - scenario = (synthesis_input.resolved_scenario or "").upper() - curated = dict(synthesis_input.curated_facts or {}) - if scenario == "EXPLAIN": - facts = dict(curated.get("explain") or {}) - must_methods = facts.get("required_methods", []) - must_calls = facts.get("required_calls", []) - must_deps = facts.get("required_dependencies", []) - must_fields = facts.get("required_fields", []) - fact_gaps = facts.get("fact_gaps", []) - return { - "answer_contract": ( - "Ответ обязан опираться на конкретные факты ниже. Если списки непусты — назови явно методы, вызовы, зависимости или поля из них. " - "Не заменяй их общими фразами. Учитывай fact_gaps." - ), - "must_mention_methods": must_methods, - "must_mention_fields": must_fields, - "must_mention_calls": must_calls, - "must_mention_dependencies": must_deps, - "must_mention_constructor_args": facts.get("required_constructor_args", []), - "must_mention_files": facts.get("required_files", []), - "must_not_infer_missing_details": True, - "fact_gaps": fact_gaps, - } - if scenario == "ARCHITECTURE": - facts = dict(curated.get("architecture") or {}) - components = facts.get("required_components", []) - relations = facts.get("required_relations", []) - verbs = facts.get("required_relation_verbs", []) - summaries = facts.get("required_relation_summaries", []) - return { - "answer_contract": ( - "Ответ обязан перечислить компоненты и связи из кода. Используй relation verbs (создаёт, вызывает, импортирует и т.д.). " - "Не подменяй архитектуру списком методов. Не используй retrieval labels в тексте." - ), - "must_mention_components": components, - "must_mention_relations": relations, - "must_mention_relation_summaries": summaries, - "must_use_relation_verbs": verbs, - "must_avoid_semantic_labels_as_primary_claims": True, - "must_not_use_retrieval_labels": facts.get("forbidden_labels", []), - "fact_gaps": facts.get("fact_gaps", []), - } - if scenario == "TRACE_FLOW": - facts = dict(curated.get("trace_flow") or {}) - steps = facts.get("required_flow_steps", []) - ordered_descriptions = facts.get("ordered_step_descriptions", []) - return { - "answer_contract": ( - "Ответ обязан описать поток как упорядоченную последовательность шагов из payload. " - "Не заявляй полноту потока, если в fact_gaps указано иное. Не делай неподтверждённых утверждений." - ), - "must_mention_flow_steps": steps, - "must_mention_ordered_steps": ordered_descriptions, - "must_mention_calls": facts.get("required_calls", []), - "must_mention_sequence_edges": facts.get("required_sequence_edges", []), - "must_avoid_overclaiming_full_flow": True, - "fact_gaps": facts.get("fact_gaps", []), - } - return {} - - def _matches_query(self, route_path: str, query_tokens: set[str]) -> bool: - if not route_path: - return False - path_tokens = self._tokens(route_path) - return bool(path_tokens & query_tokens) - - def _tokens(self, value: str) -> set[str]: - return {token.lower().strip("/") for token in _TOKEN_RE.findall(value or "") if token.strip("/")} diff --git a/src/app/modules/agent/runtime/steps/generation/prompt_selector.py b/src/app/modules/agent/runtime/steps/generation/prompt_selector.py deleted file mode 100644 index abcfeb6..0000000 --- a/src/app/modules/agent/runtime/steps/generation/prompt_selector.py +++ /dev/null @@ -1,32 +0,0 @@ -"""Выбор имени системного промпта по sub_intent и answer_mode.""" - -from __future__ import annotations - - -class RuntimePromptSelector: - _CODE_PROMPTS = { - "ARCHITECTURE": "code_qa_architecture_answer", - "EXPLAIN": "code_qa_explain_answer", - "EXPLAIN_LOCAL": "code_qa_explain_local_answer", - "FIND_ENTRYPOINTS": "code_qa_find_entrypoints_answer", - "FIND_TESTS": "code_qa_find_tests_answer", - "GENERAL_QA": "code_qa_general_answer", - "OPEN_FILE": "code_qa_open_file_answer", - "TRACE_FLOW": "code_qa_trace_flow_answer", - } - _DOCS_INTENT_PROMPTS = { - "DOCUMENTATION_EXPLAIN": "docs_explain_answer", - "GENERAL_QA": "docs_general_answer", - } - - def select(self, *, intent: str = "CODE_QA", sub_intent: str, answer_mode: str) -> str: - intent_key = (intent or "CODE_QA").upper() - if intent_key in {"OPENAPI_GENERATION", "OPENAPI_FROM_DOCUMENTATION"}: - if sub_intent.upper() == "OPENAPI_FRAGMENT_GENERATE": - return "docs_openapi_fragment_answer" - return "docs_openapi_answer" - if intent_key in self._DOCS_INTENT_PROMPTS: - return self._DOCS_INTENT_PROMPTS[intent_key] - if answer_mode in {"degraded", "not_found", "insufficient"}: - return "code_qa_degraded_answer" - return self._CODE_PROMPTS.get(sub_intent.upper(), "code_qa_explain_answer") diff --git a/src/app/modules/agent/runtime/steps/retrieval/__init__.py b/src/app/modules/agent/runtime/steps/retrieval/__init__.py deleted file mode 100644 index 7cfb2a6..0000000 --- a/src/app/modules/agent/runtime/steps/retrieval/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -"""Пакет выполнения retrieval: адаптер к RAG-репозиторию и фабрика контекста репозитория.""" - -from app.modules.agent.runtime.steps.retrieval.adapter import RuntimeRetrievalAdapter -from app.modules.agent.runtime.steps.retrieval.repo_context import RuntimeRepoContextFactory - -__all__ = ["RuntimeRetrievalAdapter", "RuntimeRepoContextFactory"] diff --git a/src/app/modules/agent/runtime/steps/retrieval/adapter.py b/src/app/modules/agent/runtime/steps/retrieval/adapter.py deleted file mode 100644 index 16d2fe7..0000000 --- a/src/app/modules/agent/runtime/steps/retrieval/adapter.py +++ /dev/null @@ -1,337 +0,0 @@ -"""Адаптер RAG-репозитория к runtime: retrieve_with_plan, retrieve_exact_files, consume_retrieval_report.""" - -from __future__ import annotations - -from time import perf_counter -from typing import TYPE_CHECKING, Any - -if TYPE_CHECKING: - from app.modules.rag.persistence.repository import RagRepository - - -class SessionEmbeddingDimensions: - def __init__(self, repository: RagRepository | None = None) -> None: - self._cache: dict[str, int] = {} - self._repository = repository - - def resolve(self, rag_session_id: str) -> int: - if rag_session_id in self._cache: - return self._cache[rag_session_id] - from app.modules.shared.db import get_engine - from sqlalchemy import text - - with get_engine().connect() as conn: - row = conn.execute( - text( - """ - SELECT vector_dims(embedding) AS dim - FROM rag_chunks - WHERE rag_session_id = :sid AND embedding IS NOT NULL - LIMIT 1 - """ - ), - {"sid": rag_session_id}, - ).mappings().first() - dim = int(row["dim"]) if row and row.get("dim") else 0 - self._cache[rag_session_id] = dim - return dim - - -class RuntimeRetrievalAdapter: - _RELATED_DOCS_THRESHOLD = 2 - - def __init__(self, repository: RagRepository | None = None) -> None: - if repository is None: - from app.modules.rag.persistence.repository import RagRepository - - repository = RagRepository() - self._repository = repository - self._dim_resolver = SessionEmbeddingDimensions(self._repository) - self._last_report: dict[str, Any] | None = None - - def retrieve_with_plan( - self, - rag_session_id: str, - query: str, - retrieval_spec, - retrieval_constraints=None, - *, - query_plan=None, - ) -> list[dict]: - rows: list[dict] = [] - planned_layers = [str(item.layer_id) for item in list(getattr(retrieval_spec, "layer_queries", []) or [])] - executed_layers: list[str] = [] - per_layer_ms: dict[str, int] = {} - layer_diagnostics: dict[str, Any] = {} - fallback_used = False - fallback_reason: str | None = None - relation_hits = 0 - query_plan_sub_intent = str(getattr(query_plan, "sub_intent", "") or "") - for layer_query in list(getattr(retrieval_spec, "layer_queries", []) or []): - layer_id = str(layer_query.layer_id) - if ( - query_plan_sub_intent == "RELATED_DOCS_EXPLAIN" - and layer_id in {"D1_DOCUMENT_CATALOG", "D0_DOC_CHUNKS"} - and relation_hits >= self._RELATED_DOCS_THRESHOLD - ): - layer_diagnostics[layer_id] = {"hits": 0, "top_ids": [], "skipped": True, "reason": "relation_primary_sufficient"} - continue - executed_layers.append(layer_id) - started = perf_counter() - layer_rows = self._retrieve_layer( - rag_session_id=rag_session_id, - query=query, - layer_id=layer_id, - limit=max(1, int(layer_query.top_k)), - path_scope=list(getattr(retrieval_spec.filters, "path_scope", []) or []), - retrieval_constraints=retrieval_constraints, - include_tests=str(getattr(retrieval_spec.filters, "test_policy", "EXCLUDE") or "EXCLUDE") == "INCLUDE", - ) - per_layer_ms[layer_id] = int((perf_counter() - started) * 1000) - layer_diagnostics[layer_id] = self._layer_diagnostics(layer_rows) - rows.extend(layer_rows) - if layer_id == "D5_RELATION_GRAPH": - relation_hits = len(layer_rows) - d2_empty = "D2_FACT_INDEX" in planned_layers and int(dict(layer_diagnostics.get("D2_FACT_INDEX") or {}).get("hits") or 0) == 0 - d0_empty = "D0_DOC_CHUNKS" in planned_layers and int(dict(layer_diagnostics.get("D0_DOC_CHUNKS") or {}).get("hits") or 0) == 0 - support_paths = self._support_paths(rows) - if support_paths and "D0_DOC_CHUNKS" in planned_layers and (d2_empty or d0_empty): - targeted_started = perf_counter() - targeted_rows = self.retrieve_exact_files( - rag_session_id, - paths=support_paths, - layers=["D0_DOC_CHUNKS"], - limit=max(6, self._planned_top_k(retrieval_spec, "D0_DOC_CHUNKS")), - query=query, - ranking_profile=str(getattr(retrieval_spec, "rerank_profile", "") or ""), - ) - merged_rows = self._dedupe([*rows, *targeted_rows]) - new_targeted_rows = self._subtract_rows(merged_rows, rows) - per_layer_ms["D0_DOC_CHUNKS"] = per_layer_ms.get("D0_DOC_CHUNKS", 0) + int((perf_counter() - targeted_started) * 1000) - rows = merged_rows - layer_diagnostics["D0_DOC_CHUNKS"] = self._layer_diagnostics( - [row for row in rows if str(row.get("layer") or "") == "D0_DOC_CHUNKS"] - ) - if new_targeted_rows: - fallback_used = True - fallback_reason = "targeted_chunk_retrieval" - self._last_report = { - "planned_layers": planned_layers, - "executed_layers": executed_layers, - "retrieval_mode_by_layer": {layer_id: "vector" for layer_id in executed_layers}, - "top_k_by_layer": {str(item.layer_id): int(item.top_k) for item in list(getattr(retrieval_spec, "layer_queries", []) or [])}, - "filters_by_layer": { - layer_id: {"path_scope": list(getattr(retrieval_spec.filters, "path_scope", []) or [])} - for layer_id in executed_layers - }, - "fallback": {"used": fallback_used, "reason": fallback_reason}, - "retrieval_by_layer_ms": per_layer_ms, - "layer_diagnostics": layer_diagnostics, - } - return self._dedupe(rows) - - def retrieve_exact_files( - self, - rag_session_id: str, - *, - repo_id: str | None = None, - paths: list[str], - layers: list[str] | None = None, - limit: int = 200, - query: str = "", - ranking_profile: str = "", - ) -> list[dict]: - started = perf_counter() - rows = self._repository.retrieve_exact_files(rag_session_id, repo_id=repo_id, paths=paths, layers=layers, limit=limit) - layer_id = list(layers or ["C0_SOURCE_CHUNKS"])[0] - self._last_report = { - "executed_layers": [layer_id], - "retrieval_mode_by_layer": {layer_id: "exact_path_fetch"}, - "top_k_by_layer": {layer_id: int(limit)}, - "filters_by_layer": {layer_id: {"path_scope": list(paths)}}, - "requests": [{"layer": layer_id, "query": query, "ranking_profile": ranking_profile}], - "fallback": {"used": False, "reason": None}, - "retrieval_by_layer_ms": {layer_id: int((perf_counter() - started) * 1000)}, - } - return self._dedupe(rows) - - def hydrate_resolved_symbol_sources( - self, - rag_session_id: str, - base_query: str, - rag_rows: list[dict], - symbol_resolution: dict, - retrieval_spec, - retrieval_constraints=None, - ) -> list[dict]: - if str(symbol_resolution.get("status") or "") != "resolved": - return rag_rows - target = str(symbol_resolution.get("resolved_symbol") or "").strip() - if not target: - return rag_rows - symbol_row = next( - ( - row - for row in rag_rows - if str(row.get("layer") or "") == "C1_SYMBOL_CATALOG" - and target in {str(row.get("title") or "").strip(), str(dict(row.get("metadata") or {}).get("qname") or "").strip()} - ), - None, - ) - if symbol_row is None: - return rag_rows - extra = self.retrieve_exact_files( - rag_session_id, - paths=[str(symbol_row.get("path") or "")], - layers=["C0_SOURCE_CHUNKS"], - limit=6, - query=base_query, - ranking_profile=str(getattr(retrieval_spec, "rerank_profile", "") or ""), - ) - return self._dedupe([*rag_rows, *extra]) - - def force_symbol_context_c0( - self, - rag_session_id: str, - *, - rag_rows: list[dict], - symbol_resolution: dict, - limit: int = 20, - ) -> list[dict]: - if str(symbol_resolution.get("status") or "") != "resolved": - return rag_rows - target = str(symbol_resolution.get("resolved_symbol") or "").strip() - target_row = next((row for row in rag_rows if str(row.get("title") or "").strip() == target), None) - if target_row is None: - return rag_rows - c0_rows = self.retrieve_exact_files( - rag_session_id, - paths=[str(target_row.get("path") or "")], - layers=["C0_SOURCE_CHUNKS"], - limit=limit, - query=target, - ranking_profile="symbol_context", - ) - preserved = [row for row in rag_rows if str(row.get("layer") or "") != "C0_SOURCE_CHUNKS"] - return self._dedupe([*preserved, *c0_rows]) - - def consume_retrieval_report(self) -> dict[str, Any] | None: - report = self._last_report - self._last_report = None - return report - - def _retrieve_layer( - self, - *, - rag_session_id: str, - query: str, - layer_id: str, - limit: int, - path_scope: list[str], - retrieval_constraints, - include_tests: bool, - ) -> list[dict]: - exclude_prefixes, exclude_like = self._test_filters(include_tests) - if retrieval_constraints is not None: - exclude_globs = [item.lower() for item in list(getattr(retrieval_constraints, "exclude_globs", []) or [])] - exclude_like.extend(self._glob_to_like(item) for item in exclude_globs if "*" in item) - dim = self._dim_resolver.resolve(rag_session_id) - if dim <= 0: - if layer_id != "C0_SOURCE_CHUNKS": - return [] - return self._repository.retrieve_lexical_code( - rag_session_id, - query_text=query, - limit=limit, - path_prefixes=path_scope or None, - exclude_path_prefixes=exclude_prefixes or None, - exclude_like_patterns=exclude_like or None, - prefer_non_tests=not include_tests, - ) - return self._repository.retrieve( - rag_session_id, - [0.0] * dim, - query_text=query, - limit=limit, - layers=[layer_id], - path_prefixes=path_scope or None, - exclude_path_prefixes=exclude_prefixes or None, - exclude_like_patterns=exclude_like or None, - prefer_non_tests=not include_tests, - ) - - def _test_filters(self, include_tests: bool) -> tuple[list[str], list[str]]: - if include_tests: - return [], [] - from app.modules.rag.retrieval.test_filter import build_test_filters - - filters = build_test_filters() - return list(filters.exclude_path_prefixes), list(filters.exclude_like_patterns) - - def _glob_to_like(self, glob: str) -> str: - return glob.replace("**/", "%/").replace("**", "%").replace("*", "%").replace("?", "_") - - def _dedupe(self, rows: list[dict]) -> list[dict]: - result: list[dict] = [] - seen: set[tuple[str, str, str, int | None, int | None]] = set() - for row in rows: - path = str(row.get("path") or "") - layer = str(row.get("layer") or "") - title = str(row.get("title") or "") - key = (layer, path, title, row.get("span_start"), row.get("span_end")) - if key in seen: - continue - seen.add(key) - result.append(row) - return result - - def _layer_diagnostics(self, rows: list[dict]) -> dict[str, Any]: - ids: list[str] = [] - sections: list[str] = [] - for row in rows[:5]: - metadata = dict(row.get("metadata") or {}) - candidate = ( - metadata.get("document_id") - or metadata.get("doc_id") - or metadata.get("fact_id") - or metadata.get("relation_id") - or metadata.get("target_id") - or row.get("path") - ) - value = str(candidate or "").strip() - if value and value not in ids: - ids.append(value) - title = str(row.get("title") or "").strip() - if title and title not in sections: - sections.append(title) - return {"hits": len(rows), "top_ids": ids, "top_sections": sections} - - def _subtract_rows(self, rows: list[dict], baseline: list[dict]) -> list[dict]: - baseline_keys = {self._row_key(row) for row in baseline} - return [row for row in rows if self._row_key(row) not in baseline_keys] - - def _support_paths(self, rows: list[dict]) -> list[str]: - values: list[str] = [] - for row in rows: - layer = str(row.get("layer") or "") - if layer not in {"D1_DOCUMENT_CATALOG", "D2_FACT_INDEX", "D3_ENTITY_CATALOG", "D4_WORKFLOW_INDEX", "D5_RELATION_GRAPH"}: - continue - path = str(row.get("path") or "").strip() - if path and path not in values: - values.append(path) - return values[:6] - - def _planned_top_k(self, retrieval_spec, layer_id: str) -> int: - for item in list(getattr(retrieval_spec, "layer_queries", []) or []): - if str(item.layer_id) == layer_id: - return int(item.top_k) - return 6 - - def _row_key(self, row: dict) -> tuple[str, str, str, int | None, int | None]: - return ( - str(row.get("layer") or ""), - str(row.get("path") or ""), - str(row.get("title") or ""), - row.get("span_start"), - row.get("span_end"), - ) diff --git a/src/app/modules/agent/runtime/steps/retrieval/repo_context.py b/src/app/modules/agent/runtime/steps/retrieval/repo_context.py deleted file mode 100644 index 5b7a163..0000000 --- a/src/app/modules/agent/runtime/steps/retrieval/repo_context.py +++ /dev/null @@ -1,40 +0,0 @@ -"""Фабрика контекста репозитория (языки, слои RAG) для runtime.""" - -from __future__ import annotations - -from app.modules.rag.contracts.enums import RagLayer -from app.modules.agent.intent_router_v2.models import RepoContext - - -class RuntimeRepoContextFactory: - _KNOWN_LAYERS = [ - RagLayer.CODE_ENTRYPOINTS, - RagLayer.CODE_SYMBOL_CATALOG, - RagLayer.CODE_DEPENDENCY_GRAPH, - RagLayer.CODE_SEMANTIC_ROLES, - RagLayer.CODE_SOURCE_CHUNKS, - RagLayer.DOCS_DOC_CHUNKS, - RagLayer.DOCS_DOCUMENT_CATALOG, - RagLayer.DOCS_FACT_INDEX, - RagLayer.DOCS_ENTITY_CATALOG, - RagLayer.DOCS_WORKFLOW_INDEX, - RagLayer.DOCS_RELATION_GRAPH, - ] - - def build(self, files_map: dict[str, dict] | None = None) -> RepoContext: - languages = self._languages(files_map or {}) - return RepoContext( - languages=languages or ["python"], - available_domains=["CODE", "DOCS"], - available_layers=list(self._KNOWN_LAYERS), - ) - - def _languages(self, files_map: dict[str, dict]) -> list[str]: - found: list[str] = [] - for path in files_map: - lower = path.lower() - if lower.endswith(".py") and "python" not in found: - found.append("python") - if lower.endswith((".md", ".rst", ".txt")) and "markdown" not in found: - found.append("markdown") - return found diff --git a/src/app/modules/agent/runtime/story_context_repository.py b/src/app/modules/agent/runtime/story_context_repository.py deleted file mode 100644 index f6bf93c..0000000 --- a/src/app/modules/agent/runtime/story_context_repository.py +++ /dev/null @@ -1,747 +0,0 @@ -"""Репозиторий Story-контекста и схемы таблиц. Перенесён из agent для отвязки от legacy-стека.""" - -from __future__ import annotations - -import json -from collections import defaultdict - -from sqlalchemy import text - -from app.modules.shared.db import get_engine - - -class StoryContextSchemaRepository: - def ensure_tables(self) -> None: - with get_engine().connect() as conn: - conn.execute( - text( - """ - CREATE TABLE IF NOT EXISTS story_records ( - story_id VARCHAR(128) PRIMARY KEY, - project_id VARCHAR(512) NOT NULL, - title TEXT NOT NULL, - status VARCHAR(64) NOT NULL DEFAULT 'draft', - owner VARCHAR(256) NULL, - metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb, - baseline_commit_sha VARCHAR(128) NULL, - snapshot_id VARCHAR(128) NULL, - created_by VARCHAR(256) NULL, - updated_by VARCHAR(256) NULL, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP - ) - """ - ) - ) - conn.execute( - text( - """ - CREATE TABLE IF NOT EXISTS story_artifacts ( - id BIGSERIAL PRIMARY KEY, - story_id VARCHAR(128) NOT NULL, - artifact_type VARCHAR(64) NOT NULL, - revision INTEGER NOT NULL, - content TEXT NOT NULL, - content_hash VARCHAR(128) NULL, - rag_session_id VARCHAR(64) NULL, - created_by VARCHAR(256) NULL, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - artifact_role VARCHAR(64) NULL, - doc_id TEXT NULL, - doc_version TEXT NULL, - path TEXT NULL, - section TEXT NULL, - chunk_id TEXT NULL, - change_type VARCHAR(32) NULL, - summary TEXT NULL, - source_ref TEXT NULL, - session_id VARCHAR(128) NULL, - CONSTRAINT fk_story_artifacts_story - FOREIGN KEY (story_id) - REFERENCES story_records(story_id) - ON DELETE CASCADE, - CONSTRAINT uq_story_artifact_revision - UNIQUE (story_id, artifact_type, revision) - ) - """ - ) - ) - conn.execute( - text( - """ - CREATE TABLE IF NOT EXISTS story_links ( - id BIGSERIAL PRIMARY KEY, - story_id VARCHAR(128) NOT NULL, - link_type VARCHAR(64) NOT NULL, - link_value TEXT NOT NULL, - target_ref TEXT NULL, - description TEXT NULL, - metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb, - created_by VARCHAR(256) NULL, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT fk_story_links_story - FOREIGN KEY (story_id) - REFERENCES story_records(story_id) - ON DELETE CASCADE - ) - """ - ) - ) - conn.execute( - text( - """ - CREATE TABLE IF NOT EXISTS session_artifacts ( - id BIGSERIAL PRIMARY KEY, - session_id VARCHAR(128) NOT NULL, - project_id VARCHAR(512) NOT NULL, - artifact_role VARCHAR(64) NOT NULL, - source_ref TEXT NULL, - doc_id TEXT NULL, - doc_version TEXT NULL, - path TEXT NULL, - section TEXT NULL, - chunk_id TEXT NULL, - change_type VARCHAR(32) NULL, - summary TEXT NULL, - created_by VARCHAR(256) NULL, - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, - bound_story_id VARCHAR(128) NULL, - bound_at TIMESTAMPTZ NULL - ) - """ - ) - ) - - # Backward-compatible schema upgrades for existing installations. - conn.execute(text("ALTER TABLE story_records ADD COLUMN IF NOT EXISTS owner VARCHAR(256) NULL")) - conn.execute(text("ALTER TABLE story_records ADD COLUMN IF NOT EXISTS metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb")) - conn.execute(text("ALTER TABLE story_records ADD COLUMN IF NOT EXISTS baseline_commit_sha VARCHAR(128) NULL")) - conn.execute(text("ALTER TABLE story_records ADD COLUMN IF NOT EXISTS snapshot_id VARCHAR(128) NULL")) - conn.execute(text("ALTER TABLE story_records ADD COLUMN IF NOT EXISTS created_by VARCHAR(256) NULL")) - conn.execute(text("ALTER TABLE story_records ADD COLUMN IF NOT EXISTS updated_by VARCHAR(256) NULL")) - - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS artifact_role VARCHAR(64) NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS doc_id TEXT NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS doc_version TEXT NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS path TEXT NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS section TEXT NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS chunk_id TEXT NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS change_type VARCHAR(32) NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS summary TEXT NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS source_ref TEXT NULL")) - conn.execute(text("ALTER TABLE story_artifacts ADD COLUMN IF NOT EXISTS session_id VARCHAR(128) NULL")) - - conn.execute(text("ALTER TABLE story_links ADD COLUMN IF NOT EXISTS target_ref TEXT NULL")) - conn.execute(text("ALTER TABLE story_links ADD COLUMN IF NOT EXISTS description TEXT NULL")) - conn.execute(text("ALTER TABLE story_links ADD COLUMN IF NOT EXISTS metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb")) - conn.execute(text("ALTER TABLE story_links ADD COLUMN IF NOT EXISTS created_by VARCHAR(256) NULL")) - - conn.execute(text("ALTER TABLE session_artifacts ADD COLUMN IF NOT EXISTS bound_story_id VARCHAR(128) NULL")) - conn.execute(text("ALTER TABLE session_artifacts ADD COLUMN IF NOT EXISTS bound_at TIMESTAMPTZ NULL")) - - conn.execute(text("CREATE INDEX IF NOT EXISTS idx_story_records_project ON story_records(project_id, updated_at DESC)")) - conn.execute(text("CREATE INDEX IF NOT EXISTS idx_story_artifacts_story_type ON story_artifacts(story_id, artifact_type, revision DESC)")) - conn.execute(text("CREATE INDEX IF NOT EXISTS idx_story_links_story ON story_links(story_id, link_type)")) - conn.execute(text("CREATE INDEX IF NOT EXISTS idx_session_artifacts_session ON session_artifacts(session_id, created_at DESC)")) - conn.execute(text("CREATE INDEX IF NOT EXISTS idx_session_artifacts_project ON session_artifacts(project_id, created_at DESC)")) - conn.commit() - - -class StoryContextRepository: - def record_story_commit( - self, - *, - story_id: str, - project_id: str, - title: str, - commit_sha: str | None, - branch: str | None, - changed_files: list[str], - summary: str, - actor: str | None, - ) -> None: - with get_engine().connect() as conn: - self._upsert_story_conn( - conn, - story_id=story_id, - project_id=project_id, - title=title, - status="in_progress", - metadata={"source": "repo_webhook"}, - baseline_commit_sha=commit_sha, - updated_by=actor, - ) - if commit_sha: - self._insert_story_link( - conn, - story_id=story_id, - link_type="commit", - target_ref=commit_sha, - description="Webhook commit", - metadata={"project_id": project_id}, - created_by=actor, - ) - if branch: - self._insert_story_link( - conn, - story_id=story_id, - link_type="external", - target_ref=branch, - description="Webhook branch", - metadata={"kind": "branch"}, - created_by=actor, - ) - for path in changed_files: - self._insert_story_link( - conn, - story_id=story_id, - link_type="doc", - target_ref=path, - description="Changed file in commit", - metadata={"commit_sha": commit_sha}, - created_by=actor, - ) - revision = self._next_revision(conn, story_id=story_id, artifact_type="commit") - conn.execute( - text( - """ - INSERT INTO story_artifacts ( - story_id, - artifact_type, - revision, - content, - content_hash, - rag_session_id, - created_by, - artifact_role, - change_type, - summary, - source_ref - ) VALUES ( - :story_id, - :artifact_type, - :revision, - :content, - NULL, - NULL, - :created_by, - :artifact_role, - :change_type, - :summary, - :source_ref - ) - """ - ), - { - "story_id": story_id, - "artifact_type": "commit", - "revision": revision, - "content": summary[:4000] or "commit", - "created_by": actor, - "artifact_role": "doc_change", - "change_type": "linked", - "summary": summary[:4000] or "commit", - "source_ref": commit_sha or branch or "webhook", - }, - ) - conn.commit() - - def upsert_story( - self, - *, - story_id: str, - project_id: str, - title: str, - status: str = "draft", - owner: str | None = None, - metadata: dict | None = None, - baseline_commit_sha: str | None = None, - snapshot_id: str | None = None, - created_by: str | None = None, - updated_by: str | None = None, - ) -> None: - with get_engine().connect() as conn: - conn.execute( - text( - """ - INSERT INTO story_records ( - story_id, - project_id, - title, - status, - owner, - metadata_json, - baseline_commit_sha, - snapshot_id, - created_by, - updated_by - ) - VALUES ( - :story_id, - :project_id, - :title, - :status, - :owner, - CAST(:metadata_json AS JSONB), - :baseline_commit_sha, - :snapshot_id, - :created_by, - :updated_by - ) - ON CONFLICT (story_id) DO UPDATE SET - project_id = EXCLUDED.project_id, - title = EXCLUDED.title, - status = EXCLUDED.status, - owner = EXCLUDED.owner, - metadata_json = EXCLUDED.metadata_json, - baseline_commit_sha = COALESCE(EXCLUDED.baseline_commit_sha, story_records.baseline_commit_sha), - snapshot_id = COALESCE(EXCLUDED.snapshot_id, story_records.snapshot_id), - updated_by = COALESCE(EXCLUDED.updated_by, story_records.updated_by), - updated_at = CURRENT_TIMESTAMP - """ - ), - { - "story_id": story_id, - "project_id": project_id, - "title": title, - "status": status, - "owner": owner, - "metadata_json": json.dumps(metadata or {}, ensure_ascii=False), - "baseline_commit_sha": baseline_commit_sha, - "snapshot_id": snapshot_id, - "created_by": created_by, - "updated_by": updated_by, - }, - ) - conn.commit() - - def add_session_artifact( - self, - *, - session_id: str, - project_id: str, - artifact_role: str, - summary: str, - change_type: str | None = None, - source_ref: str | None = None, - doc_id: str | None = None, - doc_version: str | None = None, - path: str | None = None, - section: str | None = None, - chunk_id: str | None = None, - created_by: str | None = None, - ) -> None: - with get_engine().connect() as conn: - conn.execute( - text( - """ - INSERT INTO session_artifacts ( - session_id, - project_id, - artifact_role, - source_ref, - doc_id, - doc_version, - path, - section, - chunk_id, - change_type, - summary, - created_by - ) VALUES ( - :session_id, - :project_id, - :artifact_role, - :source_ref, - :doc_id, - :doc_version, - :path, - :section, - :chunk_id, - :change_type, - :summary, - :created_by - ) - """ - ), - { - "session_id": session_id, - "project_id": project_id, - "artifact_role": artifact_role, - "source_ref": source_ref, - "doc_id": doc_id, - "doc_version": doc_version, - "path": path, - "section": section, - "chunk_id": chunk_id, - "change_type": change_type, - "summary": summary, - "created_by": created_by, - }, - ) - conn.commit() - - def bind_session_to_story( - self, - *, - session_id: str, - story_id: str, - project_id: str, - title: str, - commit_sha: str | None = None, - branch: str | None = None, - changed_files: list[str] | None = None, - actor: str | None = None, - ) -> dict: - with get_engine().connect() as conn: - self._upsert_story_conn( - conn, - story_id=story_id, - project_id=project_id, - title=title, - status="in_progress", - metadata={"bound_session_id": session_id}, - baseline_commit_sha=commit_sha, - updated_by=actor, - ) - - if commit_sha: - self._insert_story_link( - conn, - story_id=story_id, - link_type="commit", - target_ref=commit_sha, - description=f"Bound from session {session_id}", - metadata={"session_id": session_id}, - created_by=actor, - ) - if branch: - self._insert_story_link( - conn, - story_id=story_id, - link_type="external", - target_ref=branch, - description="Source branch", - metadata={"kind": "branch"}, - created_by=actor, - ) - for path in changed_files or []: - self._insert_story_link( - conn, - story_id=story_id, - link_type="doc", - target_ref=path, - description="Changed file from commit", - metadata={"session_id": session_id, "commit_sha": commit_sha}, - created_by=actor, - ) - - rows = conn.execute( - text( - """ - SELECT id, artifact_role, source_ref, doc_id, doc_version, path, section, chunk_id, change_type, summary, created_by, project_id - FROM session_artifacts - WHERE session_id = :session_id AND bound_story_id IS NULL - ORDER BY id ASC - """ - ), - {"session_id": session_id}, - ).mappings().fetchall() - - revisions = self._load_revisions(conn, story_id) - migrated = 0 - for row in rows: - role = str(row["artifact_role"]) - revisions[role] += 1 - summary = str(row["summary"] or "") - content = summary or "n/a" - conn.execute( - text( - """ - INSERT INTO story_artifacts ( - story_id, - artifact_type, - revision, - content, - content_hash, - rag_session_id, - created_by, - artifact_role, - doc_id, - doc_version, - path, - section, - chunk_id, - change_type, - summary, - source_ref, - session_id - ) VALUES ( - :story_id, - :artifact_type, - :revision, - :content, - NULL, - :rag_session_id, - :created_by, - :artifact_role, - :doc_id, - :doc_version, - :path, - :section, - :chunk_id, - :change_type, - :summary, - :source_ref, - :session_id - ) - """ - ), - { - "story_id": story_id, - "artifact_type": role, - "revision": revisions[role], - "content": content, - "rag_session_id": str(row["project_id"]), - "created_by": actor or row["created_by"], - "artifact_role": role, - "doc_id": row["doc_id"], - "doc_version": row["doc_version"], - "path": row["path"], - "section": row["section"], - "chunk_id": row["chunk_id"], - "change_type": row["change_type"], - "summary": summary, - "source_ref": row["source_ref"], - "session_id": session_id, - }, - ) - conn.execute( - text( - """ - UPDATE session_artifacts - SET bound_story_id = :story_id, bound_at = CURRENT_TIMESTAMP - WHERE id = :id - """ - ), - {"story_id": story_id, "id": int(row["id"])}, - ) - migrated += 1 - - conn.commit() - return {"story_id": story_id, "session_id": session_id, "migrated_artifacts": migrated} - - def add_artifact( - self, - *, - story_id: str, - artifact_type: str, - revision: int, - content: str, - content_hash: str | None = None, - rag_session_id: str | None = None, - created_by: str | None = None, - ) -> None: - with get_engine().connect() as conn: - conn.execute( - text( - """ - INSERT INTO story_artifacts ( - story_id, - artifact_type, - revision, - content, - content_hash, - rag_session_id, - created_by, - artifact_role, - summary - ) VALUES ( - :story_id, - :artifact_type, - :revision, - :content, - :content_hash, - :rag_session_id, - :created_by, - :artifact_role, - :summary - ) - ON CONFLICT (story_id, artifact_type, revision) DO UPDATE SET - content = EXCLUDED.content, - content_hash = EXCLUDED.content_hash, - rag_session_id = EXCLUDED.rag_session_id, - created_by = EXCLUDED.created_by, - artifact_role = EXCLUDED.artifact_role, - summary = EXCLUDED.summary - """ - ), - { - "story_id": story_id, - "artifact_type": artifact_type, - "revision": int(revision), - "content": content, - "content_hash": content_hash, - "rag_session_id": rag_session_id, - "created_by": created_by, - "artifact_role": artifact_type, - "summary": content, - }, - ) - conn.commit() - - def get_story_context(self, story_id: str) -> dict | None: - with get_engine().connect() as conn: - story = conn.execute( - text( - """ - SELECT story_id, project_id, title, status, owner, metadata_json, baseline_commit_sha, snapshot_id, created_by, updated_by, created_at, updated_at - FROM story_records - WHERE story_id = :story_id - """ - ), - {"story_id": story_id}, - ).mappings().fetchone() - if not story: - return None - - artifacts = conn.execute( - text( - """ - SELECT artifact_type, artifact_role, revision, content, summary, path, section, chunk_id, change_type, doc_id, doc_version, - source_ref, session_id, content_hash, rag_session_id, created_by, created_at - FROM story_artifacts - WHERE story_id = :story_id - ORDER BY artifact_type ASC, revision DESC - """ - ), - {"story_id": story_id}, - ).mappings().fetchall() - - links = conn.execute( - text( - """ - SELECT link_type, - COALESCE(target_ref, link_value) AS target_ref, - description, - metadata_json, - created_by, - created_at - FROM story_links - WHERE story_id = :story_id - ORDER BY id ASC - """ - ), - {"story_id": story_id}, - ).mappings().fetchall() - - return { - "story": dict(story), - "artifacts": [dict(item) for item in artifacts], - "links": [dict(item) for item in links], - } - - def _upsert_story_conn( - self, - conn, - *, - story_id: str, - project_id: str, - title: str, - status: str, - metadata: dict, - baseline_commit_sha: str | None, - updated_by: str | None, - ) -> None: - conn.execute( - text( - """ - INSERT INTO story_records (story_id, project_id, title, status, metadata_json, baseline_commit_sha, updated_by) - VALUES (:story_id, :project_id, :title, :status, CAST(:metadata_json AS JSONB), :baseline_commit_sha, :updated_by) - ON CONFLICT (story_id) DO UPDATE SET - project_id = EXCLUDED.project_id, - title = EXCLUDED.title, - status = EXCLUDED.status, - metadata_json = EXCLUDED.metadata_json, - baseline_commit_sha = COALESCE(EXCLUDED.baseline_commit_sha, story_records.baseline_commit_sha), - updated_by = COALESCE(EXCLUDED.updated_by, story_records.updated_by), - updated_at = CURRENT_TIMESTAMP - """ - ), - { - "story_id": story_id, - "project_id": project_id, - "title": title, - "status": status, - "metadata_json": json.dumps(metadata, ensure_ascii=False), - "baseline_commit_sha": baseline_commit_sha, - "updated_by": updated_by, - }, - ) - - def _insert_story_link( - self, - conn, - *, - story_id: str, - link_type: str, - target_ref: str, - description: str, - metadata: dict, - created_by: str | None, - ) -> None: - conn.execute( - text( - """ - INSERT INTO story_links (story_id, link_type, link_value, target_ref, description, metadata_json, created_by) - VALUES ( - :story_id, - :link_type, - :link_value, - :target_ref, - :description, - CAST(:metadata_json AS JSONB), - :created_by - ) - """ - ), - { - "story_id": story_id, - "link_type": link_type, - "link_value": target_ref, - "target_ref": target_ref, - "description": description, - "metadata_json": json.dumps(metadata, ensure_ascii=False), - "created_by": created_by, - }, - ) - - def _load_revisions(self, conn, story_id: str) -> defaultdict[str, int]: - rows = conn.execute( - text( - """ - SELECT artifact_type, COALESCE(MAX(revision), 0) AS max_revision - FROM story_artifacts - WHERE story_id = :story_id - GROUP BY artifact_type - """ - ), - {"story_id": story_id}, - ).mappings().fetchall() - revision_map: defaultdict[str, int] = defaultdict(int) - for row in rows: - revision_map[str(row["artifact_type"])] = int(row["max_revision"]) - return revision_map - - def _next_revision(self, conn, *, story_id: str, artifact_type: str) -> int: - row = conn.execute( - text( - """ - SELECT COALESCE(MAX(revision), 0) AS max_revision - FROM story_artifacts - WHERE story_id = :story_id AND artifact_type = :artifact_type - """ - ), - {"story_id": story_id, "artifact_type": artifact_type}, - ).mappings().fetchone() - return int(row["max_revision"]) + 1 if row else 1 diff --git a/src/app/modules/agent/task_runtime/__init__.py b/src/app/modules/agent/task_runtime/__init__.py deleted file mode 100644 index 105525c..0000000 --- a/src/app/modules/agent/task_runtime/__init__.py +++ /dev/null @@ -1,13 +0,0 @@ -from app.modules.agent.task_runtime.context import TaskRuntimeContextBuilder -from app.modules.agent.task_runtime.dispatcher import TaskWorkflowDispatcher -from app.modules.agent.task_runtime.enrichment import ContextEnrichmentService -from app.modules.agent.task_runtime.facade import AgentTaskRuntimeFacade -from app.modules.agent.task_runtime.templates import DocumentationTemplateRegistry - -__all__ = [ - "AgentTaskRuntimeFacade", - "ContextEnrichmentService", - "DocumentationTemplateRegistry", - "TaskRuntimeContextBuilder", - "TaskWorkflowDispatcher", -] diff --git a/src/app/modules/agent/task_runtime/context.py b/src/app/modules/agent/task_runtime/context.py deleted file mode 100644 index 082fe2e..0000000 --- a/src/app/modules/agent/task_runtime/context.py +++ /dev/null @@ -1,47 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.intent_router_v2 import ConversationState -from app.modules.agent.runtime.steps.retrieval import RuntimeRepoContextFactory - -from app.modules.agent.task_runtime.models import TaskRuntimeContext - - -class TaskRuntimeContextBuilder: - def __init__(self, repo_context_factory: RuntimeRepoContextFactory | None = None) -> None: - self._repo_context_factory = repo_context_factory or RuntimeRepoContextFactory() - - def build( - self, - *, - task_id: str, - dialog_session_id: str, - rag_session_id: str, - mode: str, - message: str, - attachments: list[dict], - files: list[dict], - progress_cb, - trace=None, - ) -> TaskRuntimeContext: - files_map = self._files_to_map(files) - return TaskRuntimeContext( - task_id=task_id, - dialog_session_id=dialog_session_id, - rag_session_id=rag_session_id, - mode=mode, - message=message, - attachments=list(attachments or []), - files=list(files or []), - files_map=files_map, - progress_cb=progress_cb, - trace=trace, - repo_context=self._repo_context_factory.build(files_map), - conversation_state=ConversationState(), - ) - - def _files_to_map(self, files: list[dict]) -> dict[str, dict]: - out: dict[str, dict] = {} - for item in files or []: - if isinstance(item, dict) and item.get("path"): - out[str(item["path"])] = dict(item) - return out diff --git a/src/app/modules/agent/task_runtime/dispatcher.py b/src/app/modules/agent/task_runtime/dispatcher.py deleted file mode 100644 index d891875..0000000 --- a/src/app/modules/agent/task_runtime/dispatcher.py +++ /dev/null @@ -1,37 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.task_runtime.models import TaskRuntimeContext, WorkflowExecutionResult -from app.modules.agent.task_runtime.workflows.base import TaskWorkflow - - -class TaskWorkflowDispatcher: - def __init__( - self, - *, - docs_qa: TaskWorkflow, - general_qa: TaskWorkflow, - docs_generation: TaskWorkflow, - openapi: TaskWorkflow, - fallback: TaskWorkflow, - ) -> None: - self._docs_qa = docs_qa - self._general_qa = general_qa - self._docs_generation = docs_generation - self._openapi = openapi - self._fallback = fallback - - def dispatch(self, ctx: TaskRuntimeContext) -> WorkflowExecutionResult: - workflow = self._select(getattr(ctx.route_result, "intent", "FALLBACK")) - return workflow.run(ctx) - - def _select(self, intent: str) -> TaskWorkflow: - normalized = (intent or "FALLBACK").upper() - if normalized == "DOCUMENTATION_EXPLAIN": - return self._docs_qa - if normalized == "GENERAL_QA": - return self._general_qa - if normalized == "OPENAPI_GENERATION": - return self._openapi - if normalized == "GENERATE_DOCS_FROM_CODE": - return self._docs_generation - return self._fallback diff --git a/src/app/modules/agent/task_runtime/enrichment.py b/src/app/modules/agent/task_runtime/enrichment.py deleted file mode 100644 index 7f191f6..0000000 --- a/src/app/modules/agent/task_runtime/enrichment.py +++ /dev/null @@ -1,35 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.task_runtime.models import TaskRuntimeContext - - -class AttachmentContextProvider: - def build(self, ctx: TaskRuntimeContext) -> dict[str, object]: - return {"attachments": list(ctx.attachments)} - - -class ConfluenceContextProvider: - def build(self, ctx: TaskRuntimeContext) -> dict[str, object]: - urls = [ - str(item.get("url")) - for item in ctx.attachments - if str(item.get("type") or "").lower() == "confluence_url" and item.get("url") - ] - return {"confluence_urls": urls} - - -class ContextEnrichmentService: - def __init__( - self, - attachment_provider: AttachmentContextProvider | None = None, - confluence_provider: ConfluenceContextProvider | None = None, - ) -> None: - self._attachment_provider = attachment_provider or AttachmentContextProvider() - self._confluence_provider = confluence_provider or ConfluenceContextProvider() - - def enrich(self, ctx: TaskRuntimeContext) -> dict[str, object]: - enriched: dict[str, object] = {} - enriched.update(self._attachment_provider.build(ctx)) - enriched.update(self._confluence_provider.build(ctx)) - enriched["files"] = list(ctx.files) - return enriched diff --git a/src/app/modules/agent/task_runtime/facade.py b/src/app/modules/agent/task_runtime/facade.py deleted file mode 100644 index dedd56c..0000000 --- a/src/app/modules/agent/task_runtime/facade.py +++ /dev/null @@ -1,92 +0,0 @@ -from __future__ import annotations - -import asyncio -from types import SimpleNamespace - -from app.modules.agent.intent_router_v2 import IntentRouterV2 -from app.modules.agent.task_runtime.context import TaskRuntimeContextBuilder -from app.modules.agent.task_runtime.dispatcher import TaskWorkflowDispatcher -from app.modules.agent.task_runtime.enrichment import ContextEnrichmentService -from app.modules.agent.task_runtime.status_events import emit_status_block - - -class AgentTaskRuntimeFacade: - def __init__( - self, - *, - router: IntentRouterV2, - context_builder: TaskRuntimeContextBuilder, - enrichment: ContextEnrichmentService, - dispatcher: TaskWorkflowDispatcher, - ) -> None: - self._router = router - self._context_builder = context_builder - self._enrichment = enrichment - self._dispatcher = dispatcher - - async def run( - self, - *, - task_id: str, - dialog_session_id: str, - rag_session_id: str, - mode: str, - message: str, - attachments: list[dict], - files: list[dict], - progress_cb=None, - ): - ctx = self._context_builder.build( - task_id=task_id, - dialog_session_id=dialog_session_id, - rag_session_id=rag_session_id, - mode=mode, - message=message, - attachments=attachments, - files=files, - progress_cb=progress_cb, - ) - self._notify(progress_cb, "runtime.router", "Маршрутизирую запрос по task workflows.", {"mode": mode}) - ctx.route_result = self._router.route(message, ctx.conversation_state, ctx.repo_context) - emit_status_block( - ctx, - block_id="intent_router", - title="Intent Router", - lines=[ - f"intent: {ctx.route_result.intent}", - f"sub_intent: {ctx.route_result.query_plan.sub_intent}", - f"conversation_mode: {ctx.route_result.conversation_mode}", - f"matched_source: {ctx.route_result.matched_intent_source}", - ], - ) - self._notify(progress_cb, "runtime.context", "Собираю контекст для выбранного workflow.", {"intent": ctx.route_result.intent}) - ctx.enriched_context = self._enrichment.enrich(ctx) - loop = asyncio.get_running_loop() - self._notify(progress_cb, "runtime.workflow", "Запускаю целевой task workflow.", {"intent": ctx.route_result.intent}) - emit_status_block( - ctx, - block_id="workflow", - title="Task Workflow", - lines=[ - f"intent: {ctx.route_result.intent}", - f"sub_intent: {ctx.route_result.query_plan.sub_intent}", - "dispatcher: task_workflow_dispatcher", - ], - ) - result = await loop.run_in_executor(None, lambda: self._dispatcher.dispatch(ctx)) - return SimpleNamespace( - result_type=result.result_type, - answer=result.answer, - artifacts=result.artifacts, - changeset=[], - meta={ - "task_id": task_id, - "intent": ctx.route_result.intent, - "workflow_meta": result.meta, - "route": ctx.route_result.model_dump(mode="json"), - }, - ) - - def _notify(self, progress_cb, stage: str, message: str, meta: dict) -> None: - if progress_cb is not None: - progress_cb(stage, message, "task_progress", meta) diff --git a/src/app/modules/agent/task_runtime/models.py b/src/app/modules/agent/task_runtime/models.py deleted file mode 100644 index e2ad651..0000000 --- a/src/app/modules/agent/task_runtime/models.py +++ /dev/null @@ -1,35 +0,0 @@ -from __future__ import annotations - -from dataclasses import dataclass, field -from typing import Any, Callable - -from app.schemas.chat import TaskArtifact, TaskResultType - - -ProgressCallback = Callable[[str, str, str, dict | None], Any] - - -@dataclass(slots=True) -class TaskRuntimeContext: - task_id: str - dialog_session_id: str - rag_session_id: str - mode: str - message: str - attachments: list[dict[str, Any]] = field(default_factory=list) - files: list[dict[str, Any]] = field(default_factory=list) - files_map: dict[str, dict[str, Any]] = field(default_factory=dict) - progress_cb: ProgressCallback | None = None - trace: Any = None - repo_context: Any = None - conversation_state: Any = None - route_result: Any = None - enriched_context: dict[str, Any] = field(default_factory=dict) - - -@dataclass(slots=True) -class WorkflowExecutionResult: - result_type: TaskResultType - answer: str = "" - artifacts: list[TaskArtifact] = field(default_factory=list) - meta: dict[str, Any] = field(default_factory=dict) diff --git a/src/app/modules/agent/task_runtime/status_events.py b/src/app/modules/agent/task_runtime/status_events.py deleted file mode 100644 index 34fb961..0000000 --- a/src/app/modules/agent/task_runtime/status_events.py +++ /dev/null @@ -1,28 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.task_runtime.models import TaskRuntimeContext - - -def emit_status_block( - ctx: TaskRuntimeContext, - *, - block_id: str, - title: str, - lines: list[str], - append: bool = False, -) -> None: - if ctx.progress_cb is None: - return - ctx.progress_cb( - f"status.{block_id}", - title, - "task_progress", - { - "status_block": { - "id": block_id, - "title": title, - "lines": [line for line in lines if str(line).strip()], - "append": append, - } - }, - ) diff --git a/src/app/modules/agent/task_runtime/templates.py b/src/app/modules/agent/task_runtime/templates.py deleted file mode 100644 index 2c2e46a..0000000 --- a/src/app/modules/agent/task_runtime/templates.py +++ /dev/null @@ -1,31 +0,0 @@ -from __future__ import annotations - -from dataclasses import dataclass - - -@dataclass(frozen=True, slots=True) -class DocumentationTemplate: - template_id: str - title: str - sections: tuple[str, ...] - - -class DocumentationTemplateRegistry: - def __init__(self) -> None: - self._templates = { - "system_analytics_v1": DocumentationTemplate( - template_id="system_analytics_v1", - title="Документация по системной аналитике", - sections=( - "Назначение", - "Контекст и границы", - "Основные сущности", - "Сценарии и workflow", - "Интеграции", - "Открытые вопросы и допущения", - ), - ) - } - - def default(self) -> DocumentationTemplate: - return self._templates["system_analytics_v1"] diff --git a/src/app/modules/agent/task_runtime/workflows/__init__.py b/src/app/modules/agent/task_runtime/workflows/__init__.py deleted file mode 100644 index cfd0380..0000000 --- a/src/app/modules/agent/task_runtime/workflows/__init__.py +++ /dev/null @@ -1,11 +0,0 @@ -from app.modules.agent.task_runtime.workflows.docs_generation import DocumentationGenerationWorkflow -from app.modules.agent.task_runtime.workflows.docs_qa import DocsQaWorkflow -from app.modules.agent.task_runtime.workflows.fallback import FallbackWorkflow -from app.modules.agent.task_runtime.workflows.openapi import OpenApiWorkflow - -__all__ = [ - "DocsQaWorkflow", - "DocumentationGenerationWorkflow", - "FallbackWorkflow", - "OpenApiWorkflow", -] diff --git a/src/app/modules/agent/task_runtime/workflows/base.py b/src/app/modules/agent/task_runtime/workflows/base.py deleted file mode 100644 index 625eed6..0000000 --- a/src/app/modules/agent/task_runtime/workflows/base.py +++ /dev/null @@ -1,11 +0,0 @@ -from __future__ import annotations - -from typing import Protocol - -from app.modules.agent.task_runtime.models import TaskRuntimeContext, WorkflowExecutionResult - - -class TaskWorkflow(Protocol): - workflow_id: str - - def run(self, ctx: TaskRuntimeContext) -> WorkflowExecutionResult: ... diff --git a/src/app/modules/agent/task_runtime/workflows/docs_generation.py b/src/app/modules/agent/task_runtime/workflows/docs_generation.py deleted file mode 100644 index 2d03522..0000000 --- a/src/app/modules/agent/task_runtime/workflows/docs_generation.py +++ /dev/null @@ -1,82 +0,0 @@ -from __future__ import annotations - -import json - -from app.modules.agent.llm import AgentLlmService -from app.modules.agent.task_runtime.models import TaskRuntimeContext, WorkflowExecutionResult -from app.modules.agent.task_runtime.status_events import emit_status_block -from app.modules.agent.task_runtime.templates import DocumentationTemplateRegistry -from app.schemas.chat import TaskArtifact, TaskResultType - - -class DocumentationGenerationWorkflow: - workflow_id = "docs_generation" - - def __init__( - self, - llm: AgentLlmService | None, - templates: DocumentationTemplateRegistry | None = None, - ) -> None: - self._llm = llm - self._templates = templates or DocumentationTemplateRegistry() - - def run(self, ctx: TaskRuntimeContext) -> WorkflowExecutionResult: - template = self._templates.default() - emit_status_block( - ctx, - block_id="rag_retrieval", - title="RAG Retrieval", - lines=["not used in docs_generation workflow"], - ) - answer = self._generate(ctx, template) - emit_status_block( - ctx, - block_id="workflow", - title="Task Workflow", - lines=[ - f"workflow_id: {self.workflow_id}", - f"template_id: {template.template_id}", - ], - ) - emit_status_block( - ctx, - block_id="evidence_gate", - title="Evidence Gate", - lines=["not applied in docs_generation workflow"], - ) - artifact = TaskArtifact( - artifact_type=TaskResultType.DOCUMENTATION, - title=template.title, - content=answer, - format="markdown", - template_id=template.template_id, - source_refs=list(ctx.enriched_context.get("confluence_urls") or []), - ) - return WorkflowExecutionResult( - result_type=TaskResultType.DOCUMENTATION, - answer=answer, - artifacts=[artifact], - meta={"workflow_id": self.workflow_id, "intent": getattr(ctx.route_result, "intent", "")}, - ) - - def _generate(self, ctx: TaskRuntimeContext, template) -> str: - if self._llm is None: - return self._fallback(template) - payload = json.dumps( - { - "question": ctx.message, - "template_id": template.template_id, - "title": template.title, - "sections": list(template.sections), - "attachments": list(ctx.attachments), - "files": list(ctx.files), - "context": dict(ctx.enriched_context), - }, - ensure_ascii=False, - indent=2, - ) - return self._llm.generate("docs_template_generation", payload, log_context="agent.workflow.docs_generation").strip() - - def _fallback(self, template) -> str: - sections = "\n".join(f"## {name}\n\nTBD" for name in template.sections) - return f"# {template.title}\n\n{sections}\n" diff --git a/src/app/modules/agent/task_runtime/workflows/docs_qa.py b/src/app/modules/agent/task_runtime/workflows/docs_qa.py deleted file mode 100644 index 2f55323..0000000 --- a/src/app/modules/agent/task_runtime/workflows/docs_qa.py +++ /dev/null @@ -1,93 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner -from app.modules.agent.task_runtime.models import TaskRuntimeContext, WorkflowExecutionResult -from app.modules.agent.task_runtime.status_events import emit_status_block -from app.schemas.chat import TaskResultType - - -class DocsQaWorkflow: - workflow_id = "docs_qa" - - def __init__(self, runner: DocsQAPipelineRunner) -> None: - self._runner = runner - - def run(self, ctx: TaskRuntimeContext) -> WorkflowExecutionResult: - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "started", - {"workflow_id": self.workflow_id, "message": ctx.message}, - ) - result = self._runner.run( - ctx.message, - ctx.rag_session_id, - conversation_state=ctx.conversation_state, - mode="full", - trace=ctx.trace, - ) - diagnostics = result.diagnostics.model_dump(mode="json") - emit_status_block( - ctx, - block_id="rag_retrieval", - title="RAG Retrieval", - lines=_retrieval_lines(diagnostics), - ) - emit_status_block( - ctx, - block_id="workflow", - title="Task Workflow", - lines=[ - f"workflow_id: {self.workflow_id}", - f"prompt: {result.prompt_name}", - f"answer_mode: {result.answer_mode}", - ], - ) - emit_status_block( - ctx, - block_id="evidence_gate", - title="Evidence Gate", - lines=_gate_lines(diagnostics), - ) - result_payload = WorkflowExecutionResult( - result_type=TaskResultType.ANSWER, - answer=result.answer, - meta={ - "workflow_id": self.workflow_id, - "intent": result.router_result.intent, - "diagnostics": diagnostics, - "llm_request": result.llm_request, - }, - ) - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "completed", - {"workflow_id": self.workflow_id, "answer_length": len(result.answer or "")}, - ) - return result_payload - - -def _retrieval_lines(diagnostics: dict) -> list[str]: - lines = [ - f"planned_layers: {', '.join(diagnostics.get('planned_layers') or []) or '-'}", - f"executed_layers: {', '.join(diagnostics.get('executed_layers') or []) or '-'}", - ] - layer_diagnostics = dict(diagnostics.get("layer_diagnostics") or {}) - for layer_id in diagnostics.get("executed_layers") or []: - info = dict(layer_diagnostics.get(layer_id) or {}) - hits = info.get("hits", 0) - lines.append(f"{layer_id}: {hits} hits") - return lines - - -def _gate_lines(diagnostics: dict) -> list[str]: - lines = [ - f"decision: {diagnostics.get('gate_decision') or '-'}", - f"reason: {diagnostics.get('gate_decision_reason') or '-'}", - ] - missing = list(diagnostics.get("gate_missing_requirements") or []) - satisfied = list(diagnostics.get("gate_satisfied_requirements") or []) - if missing: - lines.append(f"missing: {', '.join(missing)}") - if satisfied: - lines.append(f"satisfied: {', '.join(satisfied)}") - return lines diff --git a/src/app/modules/agent/task_runtime/workflows/fallback.py b/src/app/modules/agent/task_runtime/workflows/fallback.py deleted file mode 100644 index cb5c3b7..0000000 --- a/src/app/modules/agent/task_runtime/workflows/fallback.py +++ /dev/null @@ -1,69 +0,0 @@ -from __future__ import annotations - -import json - -from app.modules.agent.llm import AgentLlmService -from app.modules.agent.task_runtime.models import TaskRuntimeContext, WorkflowExecutionResult -from app.modules.agent.task_runtime.status_events import emit_status_block -from app.schemas.chat import TaskResultType - - -class FallbackWorkflow: - workflow_id = "fallback" - - def __init__(self, llm: AgentLlmService | None) -> None: - self._llm = llm - - def run(self, ctx: TaskRuntimeContext) -> WorkflowExecutionResult: - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "started", - {"workflow_id": self.workflow_id, "message": ctx.message}, - ) - emit_status_block( - ctx, - block_id="rag_retrieval", - title="RAG Retrieval", - lines=["not used in fallback workflow"], - ) - if self._llm is None: - answer = "Пока не удалось подобрать специализированный workflow для этого запроса." - else: - payload = json.dumps( - { - "question": ctx.message, - "attachments": list(ctx.attachments), - "confluence_urls": list(ctx.enriched_context.get("confluence_urls") or []), - }, - ensure_ascii=False, - indent=2, - ) - answer = self._llm.generate( - "fallback_answer", - payload, - log_context="agent.workflow.fallback", - trace=ctx.trace.module("llm") if ctx.trace is not None else None, - ).strip() - emit_status_block( - ctx, - block_id="workflow", - title="Task Workflow", - lines=[f"workflow_id: {self.workflow_id}"], - ) - emit_status_block( - ctx, - block_id="evidence_gate", - title="Evidence Gate", - lines=["not applied in fallback workflow"], - ) - result = WorkflowExecutionResult( - result_type=TaskResultType.ANSWER, - answer=answer, - meta={"workflow_id": self.workflow_id, "intent": getattr(ctx.route_result, "intent", "")}, - ) - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "completed", - {"workflow_id": self.workflow_id, "answer_length": len(answer)}, - ) - return result diff --git a/src/app/modules/agent/task_runtime/workflows/general_qa.py b/src/app/modules/agent/task_runtime/workflows/general_qa.py deleted file mode 100644 index 9991eb3..0000000 --- a/src/app/modules/agent/task_runtime/workflows/general_qa.py +++ /dev/null @@ -1,67 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner -from app.modules.agent.task_runtime.models import TaskRuntimeContext, WorkflowExecutionResult -from app.modules.agent.task_runtime.status_events import emit_status_block -from app.modules.agent.task_runtime.workflows.docs_qa import _gate_lines, _retrieval_lines -from app.schemas.chat import TaskResultType - - -class GeneralQaWorkflow: - workflow_id = "general_qa" - - def __init__(self, runner: DocsQAPipelineRunner) -> None: - self._runner = runner - - def run(self, ctx: TaskRuntimeContext) -> WorkflowExecutionResult: - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "started", - {"workflow_id": self.workflow_id, "message": ctx.message}, - ) - result = self._runner.run( - ctx.message, - ctx.rag_session_id, - conversation_state=ctx.conversation_state, - mode="full", - trace=ctx.trace, - ) - diagnostics = result.diagnostics.model_dump(mode="json") - emit_status_block( - ctx, - block_id="rag_retrieval", - title="RAG Retrieval", - lines=_retrieval_lines(diagnostics), - ) - emit_status_block( - ctx, - block_id="workflow", - title="Task Workflow", - lines=[ - f"workflow_id: {self.workflow_id}", - f"prompt: {result.prompt_name}", - f"answer_mode: {result.answer_mode}", - ], - ) - emit_status_block( - ctx, - block_id="evidence_gate", - title="Evidence Gate", - lines=_gate_lines(diagnostics), - ) - result_payload = WorkflowExecutionResult( - result_type=TaskResultType.ANSWER, - answer=result.answer, - meta={ - "workflow_id": self.workflow_id, - "intent": result.router_result.intent, - "diagnostics": diagnostics, - "llm_request": result.llm_request, - }, - ) - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "completed", - {"workflow_id": self.workflow_id, "answer_length": len(result.answer or "")}, - ) - return result_payload diff --git a/src/app/modules/agent/task_runtime/workflows/openapi.py b/src/app/modules/agent/task_runtime/workflows/openapi.py deleted file mode 100644 index fb6d95b..0000000 --- a/src/app/modules/agent/task_runtime/workflows/openapi.py +++ /dev/null @@ -1,76 +0,0 @@ -from __future__ import annotations - -from app.modules.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner -from app.modules.agent.task_runtime.models import TaskRuntimeContext, WorkflowExecutionResult -from app.modules.agent.task_runtime.status_events import emit_status_block -from app.schemas.chat import TaskArtifact, TaskResultType -from app.modules.agent.task_runtime.workflows.docs_qa import _gate_lines, _retrieval_lines - - -class OpenApiWorkflow: - workflow_id = "openapi_generation" - - def __init__(self, runner: DocsQAPipelineRunner) -> None: - self._runner = runner - - def run(self, ctx: TaskRuntimeContext) -> WorkflowExecutionResult: - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "started", - {"workflow_id": self.workflow_id, "message": ctx.message}, - ) - result = self._runner.run( - ctx.message, - ctx.rag_session_id, - conversation_state=ctx.conversation_state, - mode="full", - trace=ctx.trace, - ) - diagnostics = result.diagnostics.model_dump(mode="json") - emit_status_block( - ctx, - block_id="rag_retrieval", - title="RAG Retrieval", - lines=_retrieval_lines(diagnostics), - ) - emit_status_block( - ctx, - block_id="workflow", - title="Task Workflow", - lines=[ - f"workflow_id: {self.workflow_id}", - f"prompt: {result.prompt_name}", - f"answer_mode: {result.answer_mode}", - ], - ) - emit_status_block( - ctx, - block_id="evidence_gate", - title="Evidence Gate", - lines=_gate_lines(diagnostics), - ) - content = (result.openapi_result.raw_yaml if result.openapi_result else "") or result.answer - artifact = TaskArtifact( - artifact_type=TaskResultType.OPENAPI, - title="OpenAPI Specification", - content=content, - format="yaml", - source_refs=list(result.diagnostics.doc_paths), - ) - result_payload = WorkflowExecutionResult( - result_type=TaskResultType.OPENAPI, - answer=content, - artifacts=[artifact], - meta={ - "workflow_id": self.workflow_id, - "intent": result.router_result.intent, - "diagnostics": diagnostics, - "llm_request": result.llm_request, - }, - ) - if ctx.trace is not None: - ctx.trace.module("task_workflow").log( - "completed", - {"workflow_id": self.workflow_id, "answer_length": len(content or "")}, - ) - return result_payload diff --git a/src/app/modules/api/application/session_service.py b/src/app/modules/api/application/session_service.py deleted file mode 100644 index 1dfeeab..0000000 --- a/src/app/modules/api/application/session_service.py +++ /dev/null @@ -1,45 +0,0 @@ -from __future__ import annotations - -from datetime import datetime, timezone - -from app.core.exceptions import AppError -from app.modules.api.domain.models.agent_session import AgentSession -from app.modules.api.infrastructure.ids.session_id_factory import SessionIdFactory -from app.modules.api.infrastructure.stores.in_memory_session_store import InMemorySessionStore -from app.schemas.common import ModuleName - - -class SessionService: - def __init__( - self, - store: InMemorySessionStore, - ids: SessionIdFactory, - rag_session_exists, - ) -> None: - self._store = store - self._ids = ids - self._rag_session_exists = rag_session_exists - - def create(self) -> AgentSession: - session = AgentSession.create(self._ids.create()) - return self._store.save(session) - - def get(self, session_id: str) -> AgentSession: - session = self._store.get(session_id) - if session is None: - raise AppError("session_not_found", f"Agent session not found: {session_id}", ModuleName.BACKEND) - return session - - def bind_rag_session(self, session_id: str, rag_session_id: str) -> AgentSession: - if not self._rag_session_exists(rag_session_id): - raise AppError("rag_session_not_found", f"RAG session not found: {rag_session_id}", ModuleName.RAG) - session = self.get(session_id) - session.active_rag_session_id = rag_session_id - session.updated_at = datetime.now(timezone.utc) - return self._store.save(session) - - def reset(self, session_id: str) -> AgentSession: - session = self.get(session_id) - session.active_rag_session_id = None - session.updated_at = datetime.now(timezone.utc) - return self._store.save(session) diff --git a/src/app/modules/api/controllers/session_controller.py b/src/app/modules/api/controllers/session_controller.py deleted file mode 100644 index 996de49..0000000 --- a/src/app/modules/api/controllers/session_controller.py +++ /dev/null @@ -1,39 +0,0 @@ -from __future__ import annotations - -from app.schemas.agent_api import ( - BindRagSessionRequest, - BindRagSessionResponse, - CreateAgentSessionResponse, - ResetAgentSessionResponse, -) -from app.modules.api.application.session_service import SessionService - - -class SessionController: - def __init__(self, service: SessionService) -> None: - self._service = service - - def create_session(self) -> CreateAgentSessionResponse: - session = self._service.create() - return CreateAgentSessionResponse( - session_id=session.session_id, - active_rag_session_id=session.active_rag_session_id, - created_at=session.created_at, - ) - - def bind_rag_session(self, session_id: str, request: BindRagSessionRequest) -> BindRagSessionResponse: - session = self._service.bind_rag_session(session_id, request.rag_session_id) - return BindRagSessionResponse( - session_id=session.session_id, - active_rag_session_id=session.active_rag_session_id, - updated_at=session.updated_at, - ) - - def reset_session(self, session_id: str) -> ResetAgentSessionResponse: - session = self._service.reset(session_id) - return ResetAgentSessionResponse( - session_id=session.session_id, - active_rag_session_id=session.active_rag_session_id, - status="reset", - updated_at=session.updated_at, - ) diff --git a/src/app/modules/api/controllers/stream_controller.py b/src/app/modules/api/controllers/stream_controller.py deleted file mode 100644 index 1026e9d..0000000 --- a/src/app/modules/api/controllers/stream_controller.py +++ /dev/null @@ -1,37 +0,0 @@ -from __future__ import annotations - -from fastapi.responses import StreamingResponse - -from app.modules.api.application.stream_service import StreamService - - -class StreamController: - def __init__(self, service: StreamService) -> None: - self._service = service - - async def stream(self, request_id: str) -> StreamingResponse: - queue = await self._service.subscribe(request_id) - - async def event_stream(): - import asyncio - - heartbeat = 10 - try: - while True: - try: - event = await asyncio.wait_for(queue.get(), timeout=heartbeat) - yield self._service.encode(event) - except asyncio.TimeoutError: - yield ": keepalive\n\n" - finally: - await self._service.unsubscribe(request_id, queue) - - return StreamingResponse( - event_stream(), - media_type="text/event-stream", - headers={ - "Cache-Control": "no-cache, no-transform", - "Connection": "keep-alive", - "X-Accel-Buffering": "no", - }, - ) diff --git a/src/app/modules/api/domain/models/agent_session.py b/src/app/modules/api/domain/models/agent_session.py deleted file mode 100644 index 25414f0..0000000 --- a/src/app/modules/api/domain/models/agent_session.py +++ /dev/null @@ -1,22 +0,0 @@ -from __future__ import annotations - -from dataclasses import dataclass -from datetime import datetime, timezone - - -@dataclass(slots=True) -class AgentSession: - session_id: str - active_rag_session_id: str | None - created_at: datetime - updated_at: datetime - - @classmethod - def create(cls, session_id: str) -> "AgentSession": - now = datetime.now(timezone.utc) - return cls( - session_id=session_id, - active_rag_session_id=None, - created_at=now, - updated_at=now, - ) diff --git a/src/app/modules/api/infrastructure/logging/trace_file_path_builder.py b/src/app/modules/api/infrastructure/logging/trace_file_path_builder.py deleted file mode 100644 index 8bc9fd0..0000000 --- a/src/app/modules/api/infrastructure/logging/trace_file_path_builder.py +++ /dev/null @@ -1,15 +0,0 @@ -from __future__ import annotations - -from datetime import datetime, timezone -from pathlib import Path - - -class TraceFilePathBuilder: - def __init__(self, root: Path) -> None: - self._root = root - - def build(self, request_id: str) -> Path: - stamp = datetime.now(timezone.utc).strftime("%Y%m%d") - directory = self._root / stamp - directory.mkdir(parents=True, exist_ok=True) - return directory / f"{request_id}.md" diff --git a/src/app/modules/api/module.py b/src/app/modules/api/module.py deleted file mode 100644 index 9085af9..0000000 --- a/src/app/modules/api/module.py +++ /dev/null @@ -1,30 +0,0 @@ -from __future__ import annotations - -from fastapi import APIRouter - -from app.modules.api.application.request_service import RequestService -from app.modules.api.application.session_service import SessionService -from app.modules.api.application.stream_service import StreamService -from app.modules.api.controllers.request_controller import RequestController -from app.modules.api.controllers.session_controller import SessionController -from app.modules.api.controllers.stream_controller import StreamController -from app.modules.api.public_router import build_public_router - - -class ApiModule: - def __init__( - self, - sessions: SessionService, - requests: RequestService, - streams: StreamService, - ) -> None: - self._sessions = SessionController(sessions) - self._requests = RequestController(requests) - self._streams = StreamController(streams) - - def public_router(self) -> APIRouter: - return build_public_router( - sessions=self._sessions, - requests=self._requests, - streams=self._streams, - ) diff --git a/src/app/modules/application.py b/src/app/modules/application.py deleted file mode 100644 index 4d3f035..0000000 --- a/src/app/modules/application.py +++ /dev/null @@ -1,148 +0,0 @@ -from pathlib import Path - -from app.modules.agent.llm import AgentLlmService -from app.modules.agent.llm.prompt_loader import PromptLoader -from app.modules.agent.intent_router_v2 import IntentRouterV2 -from app.modules.agent.runtime import DocsQAPipelineRunner -from app.modules.agent.runtime.steps.retrieval import RuntimeRepoContextFactory, RuntimeRetrievalAdapter -from app.modules.agent.task_runtime import ContextEnrichmentService, DocumentationTemplateRegistry, TaskRuntimeContextBuilder -from app.modules.agent.task_runtime.workflows import ( - DocumentationGenerationWorkflow, - DocsQaWorkflow, - FallbackWorkflow, - OpenApiWorkflow, -) -from app.modules.agent.task_runtime.workflows.general_qa import GeneralQaWorkflow -from app.modules.api.module import ApiModule -from app.modules.api.application.request_service import RequestService -from app.modules.api.application.session_service import SessionService -from app.modules.api.application.stream_service import StreamService -from app.modules.api.infrastructure.ids.request_id_factory import RequestIdFactory -from app.modules.api.infrastructure.ids.session_id_factory import SessionIdFactory -from app.modules.api.infrastructure.logging.request_trace_logger import RequestTraceLogger -from app.modules.api.infrastructure.stores.in_memory_request_store import InMemoryRequestStore -from app.modules.api.infrastructure.stores.in_memory_session_store import InMemorySessionStore -from app.modules.api.infrastructure.streaming.sse_event_channel import SseEventChannel -from app.modules.agent.orchestration.facade import OrchestrationFacade -from app.modules.agent.orchestration.adapters.intent_router_adapter import IntentRouterAdapter -from app.modules.agent.orchestration.adapters.llm_chat_adapter import LlmChatAdapter -from app.modules.agent.orchestration.messaging.client_message_publisher import ClientMessagePublisher -from app.modules.agent.orchestration.processes.registry import ProcessRegistry -from app.modules.agent.orchestration.processes.v1.process import V1Process -from app.modules.agent.orchestration.processes.v1.prompt_payload_builder import V1PromptPayloadBuilder -from app.modules.agent.orchestration.processes.v1.simple_llm_workflow import SimpleLlmWorkflow -from app.modules.agent.orchestration.processes.v1.steps.bootstrap_step import BootstrapStep -from app.modules.agent.orchestration.processes.v1.steps.execute_llm_workflow_step import ExecuteLlmWorkflowStep -from app.modules.agent.orchestration.processes.v1.steps.finalize_step import FinalizeStep -from app.modules.agent.orchestration.processes.v2.prompt_payload_builder import V2PromptPayloadBuilder -from app.modules.agent.orchestration.processes.v2.prompt_selector import V2PromptSelector -from app.modules.agent.orchestration.processes.v2.process import V2Process -from app.modules.agent.orchestration.processes.v2.steps.execute_documentation_workflow_step import ExecuteDocumentationWorkflowStep -from app.modules.agent.orchestration.processes.v2.steps.execute_fallback_workflow_step import ExecuteFallbackWorkflowStep -from app.modules.agent.orchestration.processes.v2.steps.execute_general_qa_workflow_step import ExecuteGeneralQaWorkflowStep -from app.modules.agent.orchestration.processes.v2.steps.execute_openapi_workflow_step import ExecuteOpenApiWorkflowStep -from app.modules.agent.orchestration.processes.v2.steps.route_intent_step import RouteIntentStep -from app.modules.agent.orchestration.runtime.process_runner import ProcessRunner -from app.modules.rag.persistence.repository import RagRepository -from app.modules.agent.runtime.story_context_repository import StoryContextRepository, StoryContextSchemaRepository -from app.modules.rag.module import RagModule, RagRepoModule -from app.modules.shared.bootstrap import bootstrap_database -from app.modules.shared.event_bus import EventBus -from app.modules.shared.retry_executor import RetryExecutor - - -class ModularApplication: - def __init__(self) -> None: - self.events = EventBus() - self.retry = RetryExecutor() - self.rag_repository = RagRepository() - self.story_context_schema_repository = StoryContextSchemaRepository() - self.story_context_repository = StoryContextRepository() - - self.rag = RagModule(event_bus=self.events, retry=self.retry, repository=self.rag_repository) - self.rag_repo = RagRepoModule( - story_context_repository=self.story_context_repository, - rag_repository=self.rag_repository, - ) - from app.modules.shared.gigachat.client import GigaChatClient - from app.modules.shared.gigachat.settings import GigaChatSettings - from app.modules.shared.gigachat.token_provider import GigaChatTokenProvider - - _giga_settings = GigaChatSettings.from_env() - _giga_client = GigaChatClient(_giga_settings, GigaChatTokenProvider(_giga_settings)) - _v1_prompt_loader = PromptLoader(Path(__file__).resolve().parent / "agent/orchestration/processes/v1/prompts.yml") - _v2_prompt_loader = PromptLoader(Path(__file__).resolve().parent / "agent/orchestration/processes/v2/prompts.yml") - self._agent_llm_v1 = AgentLlmService(client=_giga_client, prompts=_v1_prompt_loader) - self._agent_llm_v2 = AgentLlmService(client=_giga_client, prompts=_v2_prompt_loader) - _router = IntentRouterV2() - _retrieval = RuntimeRetrievalAdapter(self.rag_repository) - _repo_context_factory = RuntimeRepoContextFactory() - _docs_runner = DocsQAPipelineRunner( - router=_router, - retrieval_adapter=_retrieval, - repo_context=_repo_context_factory.build(), - llm=self._agent_llm_v2, - prompt_selector=V2PromptSelector(), - prompt_payload_builder=V2PromptPayloadBuilder(), - ) - _task_context_builder = TaskRuntimeContextBuilder(_repo_context_factory) - _context_enrichment = ContextEnrichmentService() - _docs_workflow = DocsQaWorkflow(_docs_runner) - _openapi_workflow = OpenApiWorkflow(_docs_runner) - _general_qa_workflow = GeneralQaWorkflow(_docs_runner) - _fallback_workflow = FallbackWorkflow(self._agent_llm_v2) - _docs_generation_workflow = DocumentationGenerationWorkflow(self._agent_llm_v2, DocumentationTemplateRegistry()) - self._docs_generation_workflow = _docs_generation_workflow - - self.agent_sessions = InMemorySessionStore() - self.agent_requests = InMemoryRequestStore() - self.agent_events = SseEventChannel() - self.agent_trace_logger = RequestTraceLogger(Path("runtime_traces/agent_requests")) - _publisher = ClientMessagePublisher(self.agent_events, self.agent_trace_logger) - _v1_workflow = SimpleLlmWorkflow( - LlmChatAdapter(self._agent_llm_v1, prompt_name="simple_llm_answer"), - V1PromptPayloadBuilder(), - ) - _process_registry = ProcessRegistry( - V1Process([BootstrapStep(), ExecuteLlmWorkflowStep(_v1_workflow), FinalizeStep()]), - V2Process( - [ - BootstrapStep(), - RouteIntentStep(IntentRouterAdapter(_router), _task_context_builder, _context_enrichment), - ExecuteDocumentationWorkflowStep(_docs_workflow, "workflow_documentation_explain"), - ExecuteOpenApiWorkflowStep(_openapi_workflow, "workflow_openapi_generation"), - ExecuteGeneralQaWorkflowStep(_general_qa_workflow, "workflow_general_qa"), - ExecuteFallbackWorkflowStep(_fallback_workflow, "workflow_fallback"), - FinalizeStep(), - ] - ), - ) - _orchestration = OrchestrationFacade( - request_store=self.agent_requests, - process_registry=_process_registry, - process_runner=ProcessRunner(), - publisher=_publisher, - trace_logger=self.agent_trace_logger, - ) - _session_service = SessionService( - store=self.agent_sessions, - ids=SessionIdFactory(), - rag_session_exists=lambda rag_session_id: self.rag.sessions.get(rag_session_id) is not None, - ) - _request_service = RequestService( - request_store=self.agent_requests, - request_ids=RequestIdFactory(), - sessions=_session_service, - orchestration=_orchestration, - ) - self.api = ApiModule( - sessions=_session_service, - requests=_request_service, - streams=StreamService(self.agent_events, request_exists=lambda request_id: self.agent_requests.get(request_id) is not None), - ) - - def startup(self) -> None: - bootstrap_database( - self.rag_repository, - self.story_context_schema_repository, - ) diff --git a/src/app/modules/rag/contracts/__init__.py b/src/app/modules/rag/contracts/__init__.py deleted file mode 100644 index cdb52d6..0000000 --- a/src/app/modules/rag/contracts/__init__.py +++ /dev/null @@ -1,17 +0,0 @@ -from app.modules.rag.contracts.documents import RagDocument, RagSource, RagSpan -from app.modules.rag.contracts.enums import DocKind, EvidenceType, RagLayer, RetrievalMode -from app.modules.rag.contracts.evidence import EvidenceLink -from app.modules.rag.contracts.retrieval import RetrievalItem, RetrievalQuery - -__all__ = [ - "DocKind", - "EvidenceLink", - "EvidenceType", - "RagDocument", - "RagLayer", - "RagSource", - "RagSpan", - "RetrievalItem", - "RetrievalMode", - "RetrievalQuery", -] diff --git a/src/app/modules/rag/indexing/code/roles/__init__.py b/src/app/modules/rag/indexing/code/roles/__init__.py deleted file mode 100644 index f4c5b12..0000000 --- a/src/app/modules/rag/indexing/code/roles/__init__.py +++ /dev/null @@ -1,9 +0,0 @@ -from app.modules.rag.indexing.code.roles.builder import SemanticRoleBuilder -from app.modules.rag.indexing.code.roles.document_builder import SemanticRoleDocumentBuilder -from app.modules.rag.indexing.code.roles.models import SemanticRoleRecord - -__all__ = [ - "SemanticRoleBuilder", - "SemanticRoleDocumentBuilder", - "SemanticRoleRecord", -] diff --git a/src/app/modules/rag/indexing/docs/content_parser.py b/src/app/modules/rag/indexing/docs/content_parser.py deleted file mode 100644 index f5506db..0000000 --- a/src/app/modules/rag/indexing/docs/content_parser.py +++ /dev/null @@ -1,58 +0,0 @@ -from __future__ import annotations - -from app.modules.rag.indexing.docs.models import ParsedDocument, ParsedSubsection, ParsedTopLevelSection - - -class DocsContentParser: - def parse(self, frontmatter: dict, body: str, *, fallback_title: str) -> ParsedDocument: - top_sections = self._top_level_sections(body) - summary = self._section_text(top_sections, "Summary") - details = self._section_text(top_sections, "Details") - title = str(frontmatter.get("title") or "").strip() or fallback_title - detail_sections = self._subsections(details) - return ParsedDocument( - frontmatter=frontmatter, - title=title, - summary=summary, - details=details, - body=body.strip(), - detail_sections=detail_sections, - ) - - def _top_level_sections(self, body: str) -> list[ParsedTopLevelSection]: - sections: list[ParsedTopLevelSection] = [] - current_title = "" - current_lines: list[str] = [] - for line in body.splitlines(): - if line.startswith("# "): - if current_title: - sections.append(ParsedTopLevelSection(current_title, "\n".join(current_lines).strip())) - current_title = line[2:].strip() - current_lines = [] - continue - current_lines.append(line) - if current_title: - sections.append(ParsedTopLevelSection(current_title, "\n".join(current_lines).strip())) - return sections - - def _section_text(self, sections: list[ParsedTopLevelSection], title: str) -> str: - for section in sections: - if section.title.strip().lower() == title.lower(): - return section.content.strip() - return "" - - def _subsections(self, text: str) -> list[ParsedSubsection]: - sections: list[ParsedSubsection] = [] - current_title = "" - current_lines: list[str] = [] - for line in text.splitlines(): - if line.startswith("## "): - if current_title: - sections.append(ParsedSubsection(current_title, "\n".join(current_lines).strip())) - current_title = line[3:].strip() - current_lines = [] - continue - current_lines.append(line) - if current_title: - sections.append(ParsedSubsection(current_title, "\n".join(current_lines).strip())) - return sections diff --git a/src/app/modules/rag/indexing/docs/document_builder.py b/src/app/modules/rag/indexing/docs/document_builder.py deleted file mode 100644 index 80445bd..0000000 --- a/src/app/modules/rag/indexing/docs/document_builder.py +++ /dev/null @@ -1,153 +0,0 @@ -from __future__ import annotations - -from hashlib import sha256 - -from app.modules.rag.contracts import EvidenceLink, EvidenceType, RagDocument, RagLayer, RagSource -from app.modules.rag.indexing.docs.chunkers.markdown_chunker import SectionChunk -from app.modules.rag.indexing.docs.models import RelationEdge, WorkflowScenario - - -class DocsDocumentBuilder: - def build_document_catalog(self, source: RagSource, frontmatter: dict, summary_text: str, doc_kind: str, *, fallback_title: str) -> RagDocument: - document_id = str(frontmatter.get("id") or source.path).strip() - metadata = { - "document_id": document_id, - "type": str(frontmatter.get("type") or "").strip(), - "name": str(frontmatter.get("name") or "").strip(), - "title": str(frontmatter.get("title") or "").strip() or fallback_title, - "module": str(frontmatter.get("module") or frontmatter.get("domain") or "").strip(), - "layer": str(frontmatter.get("layer") or "").strip(), - "status": frontmatter.get("status"), - "updated_at": frontmatter.get("updated_at"), - "tags": frontmatter.get("tags") or [], - "entities": frontmatter.get("entities") or [], - "parent": frontmatter.get("parent"), - "children": frontmatter.get("children") or [], - "links": frontmatter.get("links") or [], - "source_path": source.path, - "summary_text": summary_text[:4000], - "doc_kind": doc_kind, - } - return RagDocument( - layer=RagLayer.DOCS_DOCUMENT_CATALOG, - source=source, - title=metadata["title"] or document_id, - text=summary_text[:4000] or metadata["title"] or document_id, - metadata=metadata, - ) - - def build_doc_chunk(self, source: RagSource, chunk: SectionChunk, frontmatter: dict, doc_kind: str) -> RagDocument: - document_id = str(frontmatter.get("id") or source.path) - metadata = { - "document_id": document_id, - "type": frontmatter.get("type"), - "module": frontmatter.get("module") or frontmatter.get("domain"), - "tags": frontmatter.get("tags") or [], - "section_path": chunk.section_path, - "section_title": chunk.section_title, - "order": chunk.order, - "doc_kind": doc_kind, - "source_path": source.path, - "artifact_type": "DOCS", - } - return RagDocument( - layer=RagLayer.DOCS_DOC_CHUNKS, - source=source, - title=f"{document_id}:{chunk.section_title}", - text=chunk.content, - metadata=metadata, - links=[EvidenceLink(type=EvidenceType.DOC_SECTION, target_id=document_id, path=source.path, note=chunk.section_path)], - ) - - def build_entity_record(self, source: RagSource, frontmatter: dict, entity: str) -> RagDocument: - document_id = str(frontmatter.get("id") or source.path) - metadata = { - "entity_name": entity, - "document_id": document_id, - "document_type": frontmatter.get("type"), - "module": frontmatter.get("module") or frontmatter.get("domain"), - "tags": frontmatter.get("tags") or [], - "source_path": source.path, - } - return RagDocument( - layer=RagLayer.DOCS_ENTITY_CATALOG, - source=source, - title=entity, - text=entity, - metadata=metadata, - ) - - def build_fact( - self, - source: RagSource, - *, - subject_id: str, - predicate: str, - obj: str, - object_ref: str | None, - anchor: str, - tags: list[str] | None = None, - ) -> RagDocument: - fact_id = sha256(f"{subject_id}|{predicate}|{obj}|{source.path}|{anchor}".encode("utf-8")).hexdigest() - metadata = { - "fact_id": fact_id, - "subject_id": subject_id, - "predicate": predicate, - "object": obj, - "object_ref": object_ref, - "anchor": anchor, - "tags": tags or [], - "source_path": source.path, - } - return RagDocument( - layer=RagLayer.DOCS_FACT_INDEX, - source=source, - title=f"{subject_id}:{predicate}", - text=f"{subject_id} {predicate} {obj}".strip(), - metadata=metadata, - links=[EvidenceLink(type=EvidenceType.DOC_FACT, target_id=fact_id, path=source.path, note=anchor)], - ) - - def build_workflow_record(self, source: RagSource, frontmatter: dict, workflow: WorkflowScenario) -> RagDocument: - document_id = str(frontmatter.get("id") or source.path) - workflow_id = sha256(f"{document_id}|workflow|{workflow.name}|{workflow.anchor}".encode("utf-8")).hexdigest() - metadata = { - "workflow_id": workflow_id, - "document_id": document_id, - "workflow_name": workflow.name, - "preconditions": workflow.preconditions, - "trigger": workflow.trigger, - "main_flow": workflow.main_flow, - "alternative_flow": workflow.alternative_flow, - "error_handling": workflow.error_handling, - "postconditions": workflow.postconditions, - "source_path": source.path, - } - text = "\n".join( - [workflow.name, *workflow.preconditions, *workflow.trigger, *workflow.main_flow, *workflow.alternative_flow] - ).strip() - return RagDocument( - layer=RagLayer.DOCS_WORKFLOW_INDEX, - source=source, - title=workflow.name, - text=text[:4000], - metadata=metadata, - ) - - def build_relation_record(self, source: RagSource, edge: RelationEdge) -> RagDocument: - relation_id = sha256(f"{edge.source_id}|{edge.relation_type}|{edge.target_id}|{source.path}|{edge.anchor}".encode("utf-8")).hexdigest() - metadata = { - "relation_id": relation_id, - "source_id": edge.source_id, - "relation_type": edge.relation_type, - "target_id": edge.target_id, - "anchor": edge.anchor, - "source_path": source.path, - } - return RagDocument( - layer=RagLayer.DOCS_RELATION_GRAPH, - source=source, - title=f"{edge.source_id}:{edge.relation_type}", - text=f"{edge.source_id} {edge.relation_type} {edge.target_id}", - metadata=metadata, - ) diff --git a/src/app/modules/rag/indexing/docs/relation_extractor.py b/src/app/modules/rag/indexing/docs/relation_extractor.py deleted file mode 100644 index 9d90853..0000000 --- a/src/app/modules/rag/indexing/docs/relation_extractor.py +++ /dev/null @@ -1,23 +0,0 @@ -from __future__ import annotations - -from app.modules.rag.indexing.docs.models import RelationEdge - - -class DocsRelationExtractor: - def extract(self, frontmatter: dict, *, source_id: str) -> list[RelationEdge]: - edges: list[RelationEdge] = [] - parent = str(frontmatter.get("parent") or "").strip() - if parent: - edges.append(RelationEdge(source_id=source_id, relation_type="parent", target_id=parent, anchor="frontmatter.parent")) - for child in frontmatter.get("children") or []: - target = str(child or "").strip() - if target: - edges.append(RelationEdge(source_id=source_id, relation_type="child", target_id=target, anchor="frontmatter.children")) - for item in frontmatter.get("links") or []: - if not isinstance(item, dict): - continue - relation_type = str(item.get("type") or "").strip() - target = str(item.get("target") or "").strip() - if relation_type and target: - edges.append(RelationEdge(source_id=source_id, relation_type=relation_type, target_id=target, anchor="frontmatter.links")) - return edges diff --git a/src/app/modules/rag/module.py b/src/app/modules/rag/module.py deleted file mode 100644 index 541ab9e..0000000 --- a/src/app/modules/rag/module.py +++ /dev/null @@ -1,294 +0,0 @@ -"""Публичный модуль RAG: сессии, индексация, API и webhook репозиториев.""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from fastapi import APIRouter, Request -from fastapi.responses import JSONResponse, StreamingResponse - -from app.core.exceptions import AppError -from app.modules.rag.embedding.gigachat_embedder import GigaChatEmbedder -from app.modules.rag.indexing_service import IndexingOrchestrator -from app.modules.rag.job_store import IndexJobStore -from app.modules.rag.persistence.repository import RagRepository -from app.modules.rag.retrieval.chunker import TextChunker -from app.modules.rag.session_store import RagSessionStore -from app.modules.rag.services.rag_service import RagService -from app.modules.rag.webhook_service import RepoWebhookService -from app.modules.shared.event_bus import EventBus -from app.modules.shared.gigachat.client import GigaChatClient -from app.modules.shared.gigachat.settings import GigaChatSettings -from app.modules.shared.gigachat.token_provider import GigaChatTokenProvider -from app.modules.shared.retry_executor import RetryExecutor -from app.schemas.common import ModuleName -from app.schemas.indexing import ( - IndexChangesRequest, - IndexJobQueuedResponse, - IndexJobResponse, - IndexSnapshotRequest, -) -from app.schemas.rag_sessions import ( - RagSessionChangesRequest, - RagSessionCreateRequest, - RagSessionCreateResponse, - RagSessionJobResponse, -) - -if TYPE_CHECKING: - from app.modules.agent.runtime.story_context_repository import StoryContextRepository - - -class RagModule: - def __init__(self, event_bus: EventBus, retry: RetryExecutor, repository: RagRepository) -> None: - self._events = event_bus - self.repository = repository - settings = GigaChatSettings.from_env() - token_provider = GigaChatTokenProvider(settings) - client = GigaChatClient(settings, token_provider) - embedder = GigaChatEmbedder(client) - self.embedder = embedder - self.rag = RagService(embedder=embedder, repository=repository, chunker=TextChunker()) - self.sessions = RagSessionStore(repository) - self.jobs = IndexJobStore(repository) - self.indexing = IndexingOrchestrator( - store=self.jobs, - rag=self.rag, - events=event_bus, - retry=retry, - ) - - def public_router(self) -> APIRouter: - router = APIRouter(tags=["rag"]) - - @router.post("/api/rag/sessions", response_model=RagSessionCreateResponse) - async def create_rag_session(request: RagSessionCreateRequest) -> RagSessionCreateResponse: - session = self.sessions.create(request.project_id) - job = await self.indexing.enqueue_snapshot( - rag_session_id=session.rag_session_id, - files=[x.model_dump() for x in request.files], - ) - return RagSessionCreateResponse( - rag_session_id=session.rag_session_id, - index_job_id=job.index_job_id, - status=job.status, - ) - - @router.post("/api/rag/sessions/{rag_session_id}/changes", response_model=IndexJobQueuedResponse) - async def rag_session_changes( - rag_session_id: str, - request: RagSessionChangesRequest, - ) -> IndexJobQueuedResponse: - session = self.sessions.get(rag_session_id) - if not session: - raise AppError("not_found", f"RAG session not found: {rag_session_id}", ModuleName.RAG) - job = await self.indexing.enqueue_changes( - rag_session_id=rag_session_id, - changed_files=[x.model_dump() for x in request.changed_files], - ) - return IndexJobQueuedResponse(index_job_id=job.index_job_id, status=job.status.value) - - @router.get("/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}", response_model=RagSessionJobResponse) - async def rag_session_job(rag_session_id: str, index_job_id: str) -> RagSessionJobResponse: - job = self.jobs.get(index_job_id) - if not job or job.rag_session_id != rag_session_id: - raise AppError("not_found", f"Index job not found: {index_job_id}", ModuleName.RAG) - return RagSessionJobResponse( - rag_session_id=rag_session_id, - index_job_id=job.index_job_id, - status=job.status, - indexed_files=job.indexed_files, - failed_files=job.failed_files, - cache_hit_files=job.cache_hit_files, - cache_miss_files=job.cache_miss_files, - error=job.error.model_dump(mode="json") if job.error else None, - ) - - @router.get("/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}/events") - async def rag_session_job_events(rag_session_id: str, index_job_id: str) -> StreamingResponse: - job = self.jobs.get(index_job_id) - if not job or job.rag_session_id != rag_session_id: - raise AppError("not_found", f"Index job not found: {index_job_id}", ModuleName.RAG) - queue = await self._events.subscribe(index_job_id, replay=True) - - async def event_stream(): - import asyncio - - heartbeat = 10 - try: - while True: - try: - event = await asyncio.wait_for(queue.get(), timeout=heartbeat) - yield EventBus.as_sse(event) - if event.name == "terminal": - break - except asyncio.TimeoutError: - yield ": keepalive\n\n" - finally: - await self._events.unsubscribe(index_job_id, queue) - - return StreamingResponse( - event_stream(), - media_type="text/event-stream", - headers={ - "Cache-Control": "no-cache, no-transform", - "Connection": "keep-alive", - "X-Accel-Buffering": "no", - }, - ) - - # Legacy compatibility endpoints. - legacy = APIRouter(prefix="/api/index", tags=["index"]) - - @legacy.post("/snapshot", response_model=IndexJobQueuedResponse) - async def index_snapshot(request: IndexSnapshotRequest) -> IndexJobQueuedResponse: - session = self.sessions.put( - rag_session_id=request.project_id, - project_id=request.project_id, - ) - job = await self.indexing.enqueue_snapshot( - rag_session_id=session.rag_session_id, - files=[x.model_dump() for x in request.files], - ) - return IndexJobQueuedResponse(index_job_id=job.index_job_id, status=job.status.value) - - @legacy.post("/changes", response_model=IndexJobQueuedResponse) - async def index_changes(request: IndexChangesRequest) -> IndexJobQueuedResponse: - rag_session_id = request.project_id - if not self.sessions.get(rag_session_id): - self.sessions.put(rag_session_id=rag_session_id, project_id=rag_session_id) - job = await self.indexing.enqueue_changes( - rag_session_id=rag_session_id, - changed_files=[x.model_dump() for x in request.changed_files], - ) - return IndexJobQueuedResponse(index_job_id=job.index_job_id, status=job.status.value) - - @legacy.get("/jobs/{index_job_id}", response_model=IndexJobResponse) - async def get_index_job(index_job_id: str) -> IndexJobResponse: - job = self.jobs.get(index_job_id) - if not job: - raise AppError("not_found", f"Index job not found: {index_job_id}", ModuleName.RAG) - return IndexJobResponse( - index_job_id=job.index_job_id, - status=job.status, - indexed_files=job.indexed_files, - failed_files=job.failed_files, - cache_hit_files=job.cache_hit_files, - cache_miss_files=job.cache_miss_files, - error=job.error, - ) - - @legacy.get("/jobs/{index_job_id}/events") - async def get_index_job_events(index_job_id: str) -> StreamingResponse: - job = self.jobs.get(index_job_id) - if not job: - raise AppError("not_found", f"Index job not found: {index_job_id}", ModuleName.RAG) - queue = await self._events.subscribe(index_job_id, replay=True) - - async def event_stream(): - import asyncio - - heartbeat = 10 - try: - while True: - try: - event = await asyncio.wait_for(queue.get(), timeout=heartbeat) - yield EventBus.as_sse(event) - if event.name == "terminal": - break - except asyncio.TimeoutError: - yield ": keepalive\n\n" - finally: - await self._events.unsubscribe(index_job_id, queue) - - return StreamingResponse( - event_stream(), - media_type="text/event-stream", - headers={ - "Cache-Control": "no-cache, no-transform", - "Connection": "keep-alive", - "X-Accel-Buffering": "no", - }, - ) - - router.include_router(legacy) - return router - - def internal_router(self) -> APIRouter: - router = APIRouter(prefix="/internal/rag", tags=["internal-rag"]) - - @router.post("/index/snapshot") - async def index_snapshot(request: IndexSnapshotRequest) -> dict: - rag_session_id = request.project_id - if not self.sessions.get(rag_session_id): - self.sessions.put(rag_session_id=rag_session_id, project_id=rag_session_id) - indexed, failed, cache_hits, cache_misses = await self.rag.index_snapshot( - rag_session_id=rag_session_id, - files=[x.model_dump() for x in request.files], - ) - return { - "indexed_files": indexed, - "failed_files": failed, - "cache_hit_files": cache_hits, - "cache_miss_files": cache_misses, - } - - @router.post("/index/changes") - async def index_changes(request: IndexChangesRequest) -> dict: - rag_session_id = request.project_id - indexed, failed, cache_hits, cache_misses = await self.rag.index_changes( - rag_session_id=rag_session_id, - changed_files=[x.model_dump() for x in request.changed_files], - ) - return { - "indexed_files": indexed, - "failed_files": failed, - "cache_hit_files": cache_hits, - "cache_miss_files": cache_misses, - } - - @router.get("/index/jobs/{index_job_id}") - async def get_job(index_job_id: str) -> dict: - job = self.jobs.get(index_job_id) - if not job: - return {"status": "not_found"} - return { - "index_job_id": job.index_job_id, - "status": job.status.value, - "indexed_files": job.indexed_files, - "failed_files": job.failed_files, - "cache_hit_files": job.cache_hit_files, - "cache_miss_files": job.cache_miss_files, - "error": job.error.model_dump(mode="json") if job.error else None, - } - - @router.post("/retrieve") - async def retrieve() -> JSONResponse: - return JSONResponse( - status_code=410, - content={ - "error": "deprecated", - "message": "POST /internal/rag/retrieve is deprecated.", - }, - ) - - return router - - -class RagRepoModule: - """Модуль webhook репозиториев (Gitea/Bitbucket).""" - - def __init__(self, story_context_repository: "StoryContextRepository", rag_repository: RagRepository) -> None: - self._webhook = RepoWebhookService(story_context_repository, rag_repository) - - def internal_router(self) -> APIRouter: - router = APIRouter(prefix="/internal/rag-repo", tags=["internal-rag-repo"]) - - @router.post("/webhook") - async def process_repo_webhook(request: Request, payload: dict) -> dict: - return self._webhook.process( - payload=payload, - headers={k: v for k, v in request.headers.items()}, - ) - - return router diff --git a/src/app/modules/rag/webhook_service.py b/src/app/modules/rag/webhook_service.py deleted file mode 100644 index 77766d0..0000000 --- a/src/app/modules/rag/webhook_service.py +++ /dev/null @@ -1,219 +0,0 @@ -"""Обработка webhook от репозиториев (Gitea, Bitbucket) для привязки коммитов к историям.""" - -from __future__ import annotations - -import re -from typing import Protocol - - -_STORY_ID_RE = re.compile(r"\b[A-Z][A-Z0-9_]*-\d+\b") - - -class StoryCommitWriter(Protocol): - def record_story_commit( - self, - *, - story_id: str, - project_id: str, - title: str, - commit_sha: str | None, - branch: str | None, - changed_files: list[str], - summary: str, - actor: str | None, - ) -> None: ... - - -class RepoCacheWriter(Protocol): - def record_repo_cache( - self, - *, - project_id: str, - commit_sha: str | None, - changed_files: list[str], - summary: str, - ) -> None: ... - - -class RepoWebhookService: - def __init__(self, story_writer: StoryCommitWriter, cache_writer: RepoCacheWriter | None = None) -> None: - self._story_writer = story_writer - self._cache_writer = cache_writer - - def process(self, *, payload: dict, provider: str | None = None, headers: dict | None = None) -> dict: - resolved_provider = self._resolve_provider(provider=provider, payload=payload, headers=headers or {}) - normalized = self._normalize(provider=resolved_provider, payload=payload) - if not normalized: - return {"accepted": False, "reason": "unsupported_or_invalid_payload"} - - cache_recorded = False - if self._cache_writer is not None: - self._cache_writer.record_repo_cache( - project_id=normalized["project_id"], - commit_sha=normalized["commit_sha"], - changed_files=normalized["changed_files"], - summary=normalized["summary"], - ) - cache_recorded = True - - story_id = self._extract_story_id(normalized["messages"]) - if not story_id: - return { - "accepted": True, - "indexed": False, - "story_bound": False, - "cache_recorded": cache_recorded, - "reason": "story_id_not_found", - } - - self._story_writer.record_story_commit( - story_id=story_id, - project_id=normalized["project_id"], - title=f"Story {story_id}", - commit_sha=normalized["commit_sha"], - branch=normalized["branch"], - changed_files=normalized["changed_files"], - summary=normalized["summary"], - actor=normalized["actor"], - ) - return { - "accepted": True, - "indexed": False, - "story_bound": True, - "cache_recorded": cache_recorded, - "story_id": story_id, - "project_id": normalized["project_id"], - "commit_sha": normalized["commit_sha"], - "changed_files": normalized["changed_files"], - } - - def _resolve_provider(self, *, provider: str | None, payload: dict, headers: dict[str, str]) -> str: - value = (provider or "").strip().lower() - if value in {"gitea", "bitbucket"}: - return value - - lowered = {str(k).lower(): str(v) for k, v in headers.items()} - if "x-gitea-event" in lowered: - return "gitea" - if "x-event-key" in lowered: - return "bitbucket" - - if isinstance(payload.get("commits"), list) and ("ref" in payload or "pusher" in payload): - return "gitea" - push = payload.get("push") - if isinstance(push, dict) and isinstance(push.get("changes"), list): - return "bitbucket" - return "" - - def _normalize(self, *, provider: str, payload: dict) -> dict | None: - key = provider.lower().strip() - if key == "gitea": - return self._normalize_gitea(payload) - if key == "bitbucket": - return self._normalize_bitbucket(payload) - return None - - def _normalize_gitea(self, payload: dict) -> dict: - repo = payload.get("repository") or {} - commits = payload.get("commits") or [] - project_id = str(repo.get("full_name") or repo.get("name") or "unknown_repo") - ref = str(payload.get("ref") or "") - branch = ref.replace("refs/heads/", "") if ref.startswith("refs/heads/") else ref or None - actor = str((payload.get("pusher") or {}).get("username") or "") or None - - messages: list[str] = [] - changed_files: set[str] = set() - commit_sha: str | None = None - for commit in commits: - if not isinstance(commit, dict): - continue - cid = str(commit.get("id") or "").strip() - if cid: - commit_sha = cid - msg = str(commit.get("message") or "").strip() - if msg: - messages.append(msg) - for key in ("added", "modified", "removed"): - for path in commit.get(key) or []: - path_value = str(path).strip() - if path_value: - changed_files.add(path_value) - - summary = messages[-1] if messages else "Webhook commit without message" - return { - "project_id": project_id, - "branch": branch, - "commit_sha": commit_sha, - "changed_files": sorted(changed_files), - "messages": messages, - "summary": summary, - "actor": actor, - } - - def _normalize_bitbucket(self, payload: dict) -> dict: - repo = payload.get("repository") or {} - project_id = str(repo.get("full_name") or repo.get("name") or "unknown_repo") - - changes = (((payload.get("push") or {}).get("changes")) or []) - messages: list[str] = [] - changed_files: set[str] = set() - commit_sha: str | None = None - branch: str | None = None - actor = None - - actor_raw = payload.get("actor") or {} - if isinstance(actor_raw, dict): - actor = str(actor_raw.get("display_name") or actor_raw.get("username") or "") or None - - for change in changes: - if not isinstance(change, dict): - continue - new_ref = change.get("new") or {} - if isinstance(new_ref, dict): - branch_name = str(new_ref.get("name") or "").strip() - if branch_name: - branch = branch_name - target = new_ref.get("target") or {} - if isinstance(target, dict): - h = str(target.get("hash") or "").strip() - if h: - commit_sha = h - msg = str(target.get("message") or "").strip() - if msg: - messages.append(msg) - - for commit in change.get("commits") or []: - if not isinstance(commit, dict): - continue - h = str(commit.get("hash") or "").strip() - if h: - commit_sha = h - msg = str(commit.get("message") or "").strip() - if msg: - messages.append(msg) - for key in ("added", "modified", "removed"): - for item in commit.get(key) or []: - if isinstance(item, dict): - path_value = str(item.get("path") or "").strip() - else: - path_value = str(item).strip() - if path_value: - changed_files.add(path_value) - - summary = messages[-1] if messages else "Webhook commit without message" - return { - "project_id": project_id, - "branch": branch, - "commit_sha": commit_sha, - "changed_files": sorted(changed_files), - "messages": messages, - "summary": summary, - "actor": actor, - } - - def _extract_story_id(self, messages: list[str]) -> str | None: - for msg in messages: - match = _STORY_ID_RE.search(msg) - if match: - return match.group(0) - return None diff --git a/src/app/modules/shared/gigachat/__init__.py b/src/app/modules/shared/gigachat/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/app/schemas/agent_api.py b/src/app/schemas/agent_api.py index a125282..20dbb87 100644 --- a/src/app/schemas/agent_api.py +++ b/src/app/schemas/agent_api.py @@ -4,30 +4,21 @@ from datetime import datetime from pydantic import BaseModel, Field +from app.schemas.indexing import FileSnapshot, IndexJobStatus from app.schemas.common import ErrorPayload +class CreateAgentSessionRequest(BaseModel): + project_id: str = Field(min_length=1) + files: list[FileSnapshot] + + class CreateAgentSessionResponse(BaseModel): session_id: str - active_rag_session_id: str | None = None - created_at: datetime - - -class BindRagSessionRequest(BaseModel): rag_session_id: str = Field(min_length=1) - - -class BindRagSessionResponse(BaseModel): - session_id: str - active_rag_session_id: str | None = None - updated_at: datetime - - -class ResetAgentSessionResponse(BaseModel): - session_id: str - active_rag_session_id: str | None = None - status: str - updated_at: datetime + index_job_id: str = Field(min_length=1) + status: IndexJobStatus + created_at: datetime class AgentRequestCreateRequest(BaseModel): diff --git a/src/app/schemas/rag_sessions.py b/src/app/schemas/rag_sessions.py index 3643150..62b2aa6 100644 --- a/src/app/schemas/rag_sessions.py +++ b/src/app/schemas/rag_sessions.py @@ -1,21 +1,6 @@ -from pydantic import BaseModel, Field +from pydantic import BaseModel -from app.schemas.indexing import ChangedFile, FileSnapshot, IndexJobStatus - - -class RagSessionCreateRequest(BaseModel): - project_id: str = Field(min_length=1) - files: list[FileSnapshot] - - -class RagSessionCreateResponse(BaseModel): - rag_session_id: str - index_job_id: str - status: IndexJobStatus - - -class RagSessionChangesRequest(BaseModel): - changed_files: list[ChangedFile] +from app.schemas.indexing import IndexJobStatus class RagSessionJobResponse(BaseModel): diff --git a/tests/__pycache__/conftest.cpython-312-pytest-9.0.2.pyc b/tests/__pycache__/conftest.cpython-312-pytest-9.0.2.pyc index 57094ca..16a4b91 100644 Binary files a/tests/__pycache__/conftest.cpython-312-pytest-9.0.2.pyc and b/tests/__pycache__/conftest.cpython-312-pytest-9.0.2.pyc differ diff --git a/tests/docs_qa_eval/fixture_adapter.py b/tests/docs_qa_eval/fixture_adapter.py index e98ccaf..177495d 100644 --- a/tests/docs_qa_eval/fixture_adapter.py +++ b/tests/docs_qa_eval/fixture_adapter.py @@ -2,11 +2,12 @@ from __future__ import annotations class InMemoryDocsRetrievalAdapter: - def __init__(self, rows: list[dict]) -> None: + def __init__(self, rows: list[dict], materialized_rows: list[dict] | None = None) -> None: self._rows = list(rows) + self._materialized_rows = list(materialized_rows or []) self._report: dict = {} - def retrieve_with_plan(self, rag_session_id: str, query: str, retrieval_spec, retrieval_constraints=None, *, query_plan=None) -> list[dict]: + def retrieve_with_plan(self, rag_session_id: str, query: str, retrieval_spec, retrieval_constraints=None, *, query_plan=None, trace=None) -> list[dict]: planned_layers = [str(item.layer_id) for item in retrieval_spec.layer_queries] query_sub_intent = str(getattr(query_plan, "sub_intent", "") or "") relation_rows = [row for row in self._rows if str(row.get("layer") or "") == "D5_RELATION_GRAPH"] @@ -60,6 +61,25 @@ class InMemoryDocsRetrievalAdapter: def retrieve_exact_files(self, rag_session_id: str, *, repo_id=None, paths: list[str], layers=None, limit: int = 200, query: str = "", ranking_profile: str = "") -> list[dict]: return [] + def materialize_document_ids(self, rag_session_id: str, *, document_ids: list[str], layers=None, limit: int = 200) -> list[dict]: + allowed_ids = {str(item).strip() for item in document_ids if str(item).strip()} + if not allowed_ids: + return [] + allowed_layers = {str(item).strip() for item in list(layers or []) if str(item).strip()} + result = [] + for row in [*self._rows, *self._materialized_rows]: + if allowed_layers and str(row.get("layer") or "") not in allowed_layers: + continue + metadata = dict(row.get("metadata") or {}) + candidates = { + str(metadata.get("document_id") or "").strip(), + str(metadata.get("doc_id") or "").strip(), + str(metadata.get("subject_id") or "").strip(), + } + if candidates & allowed_ids: + result.append(row) + return self._dedupe(result)[: max(1, int(limit))] + def hydrate_resolved_symbol_sources(self, rag_session_id: str, base_query: str, rag_rows: list[dict], symbol_resolution: dict, retrieval_spec, retrieval_constraints=None) -> list[dict]: return list(rag_rows) diff --git a/tests/docs_qa_eval/runner.py b/tests/docs_qa_eval/runner.py index 0fbc650..14cd9dd 100644 --- a/tests/docs_qa_eval/runner.py +++ b/tests/docs_qa_eval/runner.py @@ -2,8 +2,8 @@ from __future__ import annotations from dataclasses import dataclass -from app.modules.agent.intent_router_v2 import IntentRouterV2 -from app.modules.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner +from app.core.agent.intent_router import IntentRouterV2 +from app.core.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner from tests.docs_qa_eval.config import DocsEvalConfig from tests.docs_qa_eval.fixture_adapter import InMemoryDocsRetrievalAdapter from tests.docs_qa_eval.golden_loader import DocsGoldenCase, load_cases diff --git a/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml b/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml new file mode 100644 index 0000000..a9b1e8c --- /dev/null +++ b/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml @@ -0,0 +1,421 @@ +defaults: + runner: process_v2 + mode: full_chain + input: + rag_session_id: "fdf3ff03-81f0-4772-b68e-250147960243" + +cases: + - id: v2-general-01-what-is-service + query: "Что это за сервис?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + pipeline: + answer_mode: grounded_summary + llm: + non_empty: true + contains_all: ["Telegram", "сервис"] + + - id: v2-general-02-purpose + query: "Для чего нужен test_echo_app?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + pipeline: + answer_mode: grounded_summary + llm: + non_empty: true + contains_all: ["test_echo_app"] + + - id: v2-general-03-problem + query: "Какую задачу решает это приложение?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + pipeline: + answer_mode: grounded_summary + llm: + non_empty: true + contains_all: ["Telegram"] + + - id: v2-general-04-docs-contents + query: "Что входит в документацию этого проекта?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + llm: + non_empty: true + contains_all: ["документац", "архитект"] + + - id: v2-general-05-main-parts + query: "Какие основные части есть у системы?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + llm: + non_empty: true + contains_all: ["систем", "Telegram"] + + - id: v2-general-06-high-level-architecture + query: "Как в целом устроено приложение?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + llm: + non_empty: true + contains_all: ["приложен"] + + - id: v2-general-07-besides-api + query: "Что тут есть кроме API?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + llm: + non_empty: true + contains_all: ["API"] + + - id: v2-general-08-components + query: "Из чего состоит Telegram Notify App?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + llm: + non_empty: true + contains_all: ["Telegram Notify App"] + + - id: v2-general-09-reading-order + query: "Какие документы стоит читать сначала?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + llm: + non_empty: true + contains_all: ["README", "архитект"] + + - id: v2-general-10-short-summary + query: "Дай короткое summary по проектной документации" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + llm: + non_empty: true + contains_all: ["документац"] + + - id: v2-docs-summary-01-health + query: "Кратко объясни по документации, как работает `/health`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + layers_include: ["D1_DOCUMENT_CATALOG", "D3_ENTITY_CATALOG", "D0_DOC_CHUNKS"] + doc_scope_contains: ["docs/README.md", "docs/architecture/telegram-notify-app-overview.md"] + llm: + non_empty: true + contains_all: ["/health", "status", "components"] + pipeline: + answer_mode: grounded_summary + + - id: v2-docs-summary-02-send + query: "Что делает endpoint `/send`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + layers_include: ["D1_DOCUMENT_CATALOG", "D3_ENTITY_CATALOG", "D0_DOC_CHUNKS"] + doc_scope_contains: ["docs/api/send-message-endpoint.md"] + llm: + non_empty: true + contains_all: ["/send", "message", "Telegram"] + + - id: v2-docs-summary-03-actions + query: "Объясни назначение `/actions/{action}`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + doc_scope_contains: ["docs/api/control-actions-endpoint.md"] + llm: + non_empty: true + contains_all: ["/actions/{action}"] + + - id: v2-docs-summary-04-runtime-health + query: "Что такое runtime health в этой документации?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + doc_scope_contains: ["docs/domains/runtime-health-entity.md"] + llm: + non_empty: true + contains_all: ["runtime health"] + + - id: v2-docs-summary-05-notification-loop + query: "Кратко опиши цикл отправки уведомлений" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + doc_scope_contains: ["docs/logic/telegram-notification-loop.md"] + llm: + non_empty: true + contains_all: ["worker", "interval", "degraded"] + + - id: v2-docs-summary-06-architecture + query: "Как устроена архитектура Telegram Notify App?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + doc_scope_contains: ["docs/architecture/telegram-notify-app-overview.md"] + llm: + non_empty: true + contains_all: ["RuntimeManager", "TelegramControlChannel"] + + - id: v2-docs-summary-07-worker + query: "Что делает worker в этом приложении?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + doc_scope_contains: ["docs/logic/telegram-notification-loop.md", "docs/architecture/telegram-notify-app-overview.md"] + llm: + non_empty: true + contains_all: ["worker", "Telegram"] + + - id: v2-docs-summary-08-health-statuses + query: "Какие статусы здоровья есть у runtime?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + doc_scope_contains: ["docs/domains/runtime-health-entity.md", "docs/api/health-endpoint.md"] + llm: + non_empty: true + contains_all: ["ok", "degraded", "unhealthy", "unknown"] + + - id: v2-docs-summary-09-manual-send + query: "Как в приложении устроена ручная отправка сообщения?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + doc_scope_contains: ["docs/api/send-message-endpoint.md"] + llm: + non_empty: true + contains_all: ["/send", "message", "TelegramSendService"] + + - id: v2-docs-summary-10-telegram-integration + query: "Как сервис взаимодействует с Telegram в целом?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + retrieval: + non_empty: true + min_rows: 2 + doc_scope_contains: ["docs/architecture/telegram-notify-app-overview.md", "docs/api/send-message-endpoint.md"] + llm: + non_empty: true + contains_all: ["Telegram", "worker", "/send"] + + - id: v2-find-files-01-health + query: "В каком файле описан `/health`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/api/health-endpoint.md"] + + - id: v2-find-files-02-send + query: "Покажи файл с описанием endpoint `/send`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/api/send-message-endpoint.md"] + + - id: v2-find-files-03-actions + query: "Где находится документация по `/actions/{action}`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/api/control-actions-endpoint.md"] + + - id: v2-find-files-04-runtime-health + query: "В каком документе описан runtime health?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/domains/runtime-health-entity.md"] + + - id: v2-find-files-05-loop + query: "Найди файл с логикой цикла отправки уведомлений" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/logic/telegram-notification-loop.md"] + + - id: v2-find-files-06-architecture + query: "Где описана архитектура приложения?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/architecture/telegram-notify-app-overview.md"] + + - id: v2-find-files-07-errors-catalog + query: "В каком файле лежит каталог ошибок?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/errors/catalog.yaml"] + + - id: v2-find-files-08-docs-index + query: "Покажи файл-индекс документации проекта" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: ["docs/README.md"] + + - id: v2-find-files-09-telegram-bot-api + query: "Какие файлы относятся к Telegram Bot API интеграции?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + min_rows: 2 + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: + ["docs/architecture/telegram-notify-app-overview.md", "docs/api/send-message-endpoint.md", "docs/logic/telegram-notification-loop.md"] + + - id: v2-find-files-10-telegram-notify-worker + query: "В каких файлах описан worker `telegram_notify`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + retrieval: + non_empty: true + min_rows: 2 + pipeline: + answer_mode: deterministic + llm: + non_empty: true + contains_all: + ["docs/architecture/telegram-notify-app-overview.md", "docs/logic/telegram-notification-loop.md", "docs/domains/runtime-health-entity.md"] diff --git a/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml b/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml new file mode 100644 index 0000000..358fb82 --- /dev/null +++ b/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml @@ -0,0 +1,244 @@ +defaults: + runner: process_v2 + mode: router_only + +cases: + - id: v2-router-general-01-what-is-service + query: "Что это за сервис?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-02-purpose + query: "Для чего нужен test_echo_app?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-03-problem + query: "Какую задачу решает это приложение?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-04-docs-contents + query: "Что входит в документацию этого проекта?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-05-main-parts + query: "Какие основные части есть у системы?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-06-high-level-architecture + query: "Как в целом устроено приложение?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-07-besides-api + query: "Что тут есть кроме API?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-08-components + query: "Из чего состоит Telegram Notify App?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-09-reading-order + query: "Какие документы стоит читать сначала?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-10-short-summary + query: "Дай короткое summary по проектной документации" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-docs-summary-01-health + query: "Кратко объясни по документации, как работает `/health`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-02-send + query: "Что делает endpoint `/send`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-03-actions + query: "Объясни назначение `/actions/{action}`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-04-runtime-health + query: "Что такое runtime health в этой документации?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-05-notification-loop + query: "Кратко опиши цикл отправки уведомлений" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-06-architecture + query: "Как устроена архитектура Telegram Notify App?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-07-worker + query: "Что делает worker в этом приложении?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-08-health-statuses + query: "Какие статусы здоровья есть у runtime?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-09-manual-send + query: "Как в приложении устроена ручная отправка сообщения?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-10-telegram-integration + query: "Как сервис взаимодействует с Telegram в целом?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-find-files-01-health + query: "В каком файле описан `/health`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-02-send + query: "Покажи файл с описанием endpoint `/send`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-03-actions + query: "Где находится документация по `/actions/{action}`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-04-runtime-health + query: "В каком документе описан runtime health?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-05-loop + query: "Найди файл с логикой цикла отправки уведомлений" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-06-architecture + query: "Где описана архитектура приложения?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-07-errors-catalog + query: "В каком файле лежит каталог ошибок?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-08-docs-index + query: "Покажи файл-индекс документации проекта" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-09-telegram-bot-api + query: "Какие файлы относятся к Telegram Bot API интеграции?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-10-telegram-notify-worker + query: "В каких файлах описан worker `telegram_notify`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES diff --git a/tests/pipeline_setup_v3/cli/index_repo.py b/tests/pipeline_setup_v3/cli/index_repo.py index d199347..e3d9814 100644 --- a/tests/pipeline_setup_v3/cli/index_repo.py +++ b/tests/pipeline_setup_v3/cli/index_repo.py @@ -10,7 +10,7 @@ _src = _agent_root / "src" if _src.exists() and str(_src) not in sys.path: sys.path.insert(0, str(_src)) -from app.modules.rag.persistence.repository import RagRepository +from app.core.rag.persistence.repository import RagRepository from tests.pipeline_setup_v3.shared.env_loader import load_pipeline_setup_env from tests.pipeline_setup_v3.shared.rag_indexer import RagSessionIndexer diff --git a/tests/pipeline_setup_v3/core/case_loader.py b/tests/pipeline_setup_v3/core/case_loader.py index a349ee0..aac0b69 100644 --- a/tests/pipeline_setup_v3/core/case_loader.py +++ b/tests/pipeline_setup_v3/core/case_loader.py @@ -80,6 +80,7 @@ class CaseDirectoryLoader: pipeline = dict(raw.get("pipeline") or {}) return CaseExpectations( router=RouterExpectation( + domain=str(router.get("domain") or "").strip() or None, intent=str(router.get("intent") or "").strip() or None, sub_intent=str(router.get("sub_intent") or "").strip() or None, graph_id=str(router.get("graph_id") or "").strip() or None, @@ -114,4 +115,6 @@ class CaseDirectoryLoader: def _normalize_runner(self, value: str) -> str: if value in {"agent_runtime", "runtime", "code_qa_eval"}: return "agent_runtime" + if value in {"process_v2", "v2_process", "v2"}: + return "process_v2" return value diff --git a/tests/pipeline_setup_v3/core/models.py b/tests/pipeline_setup_v3/core/models.py index 5ecb506..444ef76 100644 --- a/tests/pipeline_setup_v3/core/models.py +++ b/tests/pipeline_setup_v3/core/models.py @@ -5,7 +5,7 @@ from pathlib import Path from typing import Literal -RunnerKind = Literal["agent_runtime"] +RunnerKind = Literal["agent_runtime", "process_v2"] ModeKind = Literal["router_only", "router_rag", "full_chain"] @@ -18,6 +18,7 @@ class CaseInput: @dataclass(slots=True, frozen=True) class RouterExpectation: + domain: str | None = None intent: str | None = None sub_intent: str | None = None graph_id: str | None = None diff --git a/tests/pipeline_setup_v3/core/runner.py b/tests/pipeline_setup_v3/core/runner.py index a5cdd7a..209dc6f 100644 --- a/tests/pipeline_setup_v3/core/runner.py +++ b/tests/pipeline_setup_v3/core/runner.py @@ -19,10 +19,12 @@ class V3Runner: *, pipeline_mode: str = "full", router_llm_mode: str = "deterministic", + workflow_llm_enabled: bool = True, ) -> None: self._cases_dir = cases_dir self._pipeline_mode = pipeline_mode self._router_llm_mode = router_llm_mode + self._workflow_llm_enabled = workflow_llm_enabled self._validator = CaseValidator() self._sessions = RagSessionProvider() self._agent_runtime = None @@ -50,9 +52,11 @@ class V3Runner: return results, self._writer.write_summary(results) def _execute(self, case, rag_session_id): - if case.runner != "agent_runtime": - raise ValueError(f"Unsupported runner: {case.runner}") - return self._agent_runtime_adapter().execute(case, rag_session_id) + if case.runner == "agent_runtime": + return self._agent_runtime_adapter().execute(case, rag_session_id) + if case.runner == "process_v2": + return self._v2_process_adapter().execute(case, rag_session_id) + raise ValueError(f"Unsupported runner: {case.runner}") def _agent_runtime_adapter(self): if self._agent_runtime is None: @@ -63,3 +67,10 @@ class V3Runner: router_llm_mode=self._router_llm_mode, ) return self._agent_runtime + + def _v2_process_adapter(self): + if not hasattr(self, "_v2_process"): + from tests.pipeline_setup_v3.runtime.v2_process_adapter import V2ProcessAdapter + + self._v2_process = V2ProcessAdapter(workflow_llm_enabled=self._workflow_llm_enabled) + return self._v2_process diff --git a/tests/pipeline_setup_v3/core/session_provider.py b/tests/pipeline_setup_v3/core/session_provider.py index 5b3c72b..6c11cf2 100644 --- a/tests/pipeline_setup_v3/core/session_provider.py +++ b/tests/pipeline_setup_v3/core/session_provider.py @@ -24,7 +24,7 @@ class RagSessionProvider: return self._cache[key] def _build_indexer(self): - from app.modules.rag.persistence.repository import RagRepository + from app.core.rag.persistence.repository import RagRepository from tests.pipeline_setup_v3.shared.rag_indexer import RagSessionIndexer if self._repository is None: diff --git a/tests/pipeline_setup_v3/core/validators.py b/tests/pipeline_setup_v3/core/validators.py index 82d69be..c207b0a 100644 --- a/tests/pipeline_setup_v3/core/validators.py +++ b/tests/pipeline_setup_v3/core/validators.py @@ -6,6 +6,7 @@ from tests.pipeline_setup_v3.core.models import V3Case class CaseValidator: def validate(self, case: V3Case, actual: dict, details: dict | None = None) -> list[str]: mismatches: list[str] = [] + self._check(case.expectations.router.domain, actual.get("domain"), "domain", mismatches) self._check(case.expectations.router.intent, actual.get("intent"), "intent", mismatches) self._check(case.expectations.router.sub_intent, actual.get("sub_intent"), "sub_intent", mismatches) self._check(case.expectations.router.graph_id, actual.get("graph_id"), "graph_id", mismatches) diff --git a/tests/pipeline_setup_v3/run.py b/tests/pipeline_setup_v3/run.py index 8e64e27..2f47601 100644 --- a/tests/pipeline_setup_v3/run.py +++ b/tests/pipeline_setup_v3/run.py @@ -21,6 +21,12 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument("--run-name", default="manual_run", help="Output directory prefix") parser.add_argument("--results-dir", default=None, help="Override results root directory") parser.add_argument("--pipeline-mode", default="full", choices=["full", "pre_llm_only"], help="Pipeline execution mode") + parser.add_argument( + "--workflow-llm-enabled", + default="true", + choices=["true", "false"], + help="Enable or disable final workflow LLM calls where supported", + ) parser.add_argument( "--router-llm-mode", default="deterministic", @@ -39,6 +45,7 @@ def main(argv: list[str] | None = None) -> int: run_name=str(ns.run_name).strip() or "manual_run", pipeline_mode=str(ns.pipeline_mode).strip() or "full", router_llm_mode=str(ns.router_llm_mode).strip() or "deterministic", + workflow_llm_enabled=str(ns.workflow_llm_enabled).strip().lower() == "true", ) print(f"Cases dir: {cases_dir}") print(f"Run dir: {runner.run_dir}") diff --git a/tests/pipeline_setup_v3/runtime/__init__.py b/tests/pipeline_setup_v3/runtime/__init__.py index 275f5d9..7ab0434 100644 --- a/tests/pipeline_setup_v3/runtime/__init__.py +++ b/tests/pipeline_setup_v3/runtime/__init__.py @@ -1,3 +1,4 @@ -from tests.pipeline_setup_v3.runtime.agent_runtime_adapter import AgentRuntimeAdapter - -__all__ = ["AgentRuntimeAdapter"] +__all__ = [ + "agent_runtime_adapter", + "v2_process_adapter", +] diff --git a/tests/pipeline_setup_v3/runtime/agent_runtime_adapter.py b/tests/pipeline_setup_v3/runtime/agent_runtime_adapter.py index f635f66..8c9ff8d 100644 --- a/tests/pipeline_setup_v3/runtime/agent_runtime_adapter.py +++ b/tests/pipeline_setup_v3/runtime/agent_runtime_adapter.py @@ -2,20 +2,20 @@ from __future__ import annotations import math -from app.modules.agent.runtime import ( +from app.core.agent.runtime import ( AgentRuntimeExecutor, DocsQAPipelineRunner, RuntimeRepoContextFactory, RuntimeRetrievalAdapter, ) -from app.modules.agent.llm import AgentLlmService -from app.modules.agent.llm.prompt_loader import PromptLoader -from app.modules.agent.runtime.steps.context import build_retrieval_request, build_retrieval_result -from app.modules.agent.intent_router_v2 import ConversationState, IntentRouterV2 -from app.modules.agent.intent_router_v2.factory import GigaChatIntentRouterFactory -from app.modules.shared.gigachat.client import GigaChatClient -from app.modules.shared.gigachat.settings import GigaChatSettings -from app.modules.shared.gigachat.token_provider import GigaChatTokenProvider +from app.core.agent.llm import AgentLlmService +from app.core.agent.llm.prompt_loader import PromptLoader +from app.core.agent.runtime.steps.context import build_retrieval_request, build_retrieval_result +from app.core.agent.intent_router import ConversationState, IntentRouterV2 +from app.core.agent.intent_router.factory import GigaChatIntentRouterFactory +from app.core.shared.gigachat.client import GigaChatClient +from app.core.shared.gigachat.settings import GigaChatSettings +from app.core.shared.gigachat.token_provider import GigaChatTokenProvider from tests.pipeline_setup_v3.core.models import ExecutionPayload, V3Case @@ -190,6 +190,7 @@ class AgentRuntimeAdapter: ) -> dict: route_dump = route.model_dump(mode="json") if route else {} return { + "domain": route_dump.get("routing_domain") or route_dump.get("domain") or _infer_domain(route_dump.get("intent")), "intent": route_dump.get("intent"), "sub_intent": dict(route_dump.get("query_plan") or {}).get("sub_intent"), "graph_id": route_dump.get("graph_id"), @@ -262,6 +263,7 @@ class AgentRuntimeAdapter: entity_candidates = tuple(diagnostics.query_entity_candidates or self._docs_entity_candidates(result.raw_rows)) doc_scope = tuple(diagnostics.doc_ids or self._docs_scope(result.raw_rows)) return { + "domain": getattr(route, "routing_domain", None) or getattr(route, "domain", None) or _infer_domain(route.intent), "intent": route.intent, "sub_intent": route.query_plan.sub_intent, "graph_id": route.graph_id, @@ -287,6 +289,7 @@ class AgentRuntimeAdapter: def _stubbed_code_payload(self, case: V3Case, route) -> ExecutionPayload: actual = { + "domain": _infer_domain(route.intent), "intent": route.intent, "sub_intent": route.query_plan.sub_intent, "graph_id": route.graph_id, @@ -493,3 +496,12 @@ def _input_step(query: str) -> dict: "input": {"query": query}, "output": {"query": query}, } + + +def _infer_domain(intent: str | None) -> str | None: + value = str(intent or "").strip() + if value == "GENERAL_QA": + return "GENERAL" + if value in {"DOCUMENTATION_EXPLAIN", "DOC_EXPLAIN"}: + return "DOCS" + return None diff --git a/tests/pipeline_setup_v3/runtime/v2_process_adapter.py b/tests/pipeline_setup_v3/runtime/v2_process_adapter.py new file mode 100644 index 0000000..92c22fa --- /dev/null +++ b/tests/pipeline_setup_v3/runtime/v2_process_adapter.py @@ -0,0 +1,420 @@ +from __future__ import annotations + +import asyncio +from dataclasses import asdict, dataclass +from pathlib import Path + +from app.core.agent.processes.v2.anchor_signals import route_anchor_summary +from app.core.agent.processes.v2 import V2IntentRouter +from app.core.agent.processes.v2.evidence.assembler import DocsEvidenceAssembler +from app.core.agent.processes.v2.evidence.gate import DocsEvidenceGate +from app.core.agent.processes.v2.models import RetrievedFile, RetrievedSummary, V2Intent, V2Subintent +from app.core.agent.processes.v2.retrieval import DocsMetadataLookupIndex +from app.core.agent.processes.v2.retrieval.policy_resolver import V2RetrievalPolicyResolver +from app.core.agent.processes.v2.retrieval.v2_rag_adapter import V2RagRetrievalAdapter +from app.core.agent.processes.v2.workflows.docs_explain_find_files.context import DocsExplainFindFilesContext +from app.core.agent.processes.v2.workflows.docs_explain_find_files.graph import DocsExplainFindFilesGraph +from app.core.agent.processes.v2.workflows.docs_explain_summary.context import DocsExplainSummaryContext +from app.core.agent.processes.v2.workflows.docs_explain_summary.graph import DocsExplainSummaryGraph +from app.core.agent.processes.v2.workflows.general_summary.context import GeneralSummaryContext +from app.core.agent.processes.v2.workflows.general_summary.graph import GeneralSummaryGraph +from app.core.agent.utils.llm import AgentLlmService, PromptLoader +from app.core.rag.embedding.gigachat_embedder import GigaChatEmbedder +from app.core.rag.persistence import RagRepository +from app.core.rag.retrieval.session_retriever import RagSessionRetriever +from app.core.shared.gigachat.client import GigaChatClient +from app.core.shared.gigachat.settings import GigaChatSettings +from app.core.shared.gigachat.token_provider import GigaChatTokenProvider +from app.infra.observability.module_trace import RequestTraceContext +from tests.pipeline_setup_v3.core.models import ExecutionPayload, V3Case + + +class V2ProcessAdapter: + def __init__(self, *, workflow_llm_enabled: bool = True) -> None: + self._workflow_llm_enabled = workflow_llm_enabled + self._router = V2IntentRouter(llm=_build_v2_llm()) + self._policy = V2RetrievalPolicyResolver() + retriever = RagSessionRetriever(repository=RagRepository(), embedder=GigaChatEmbedder(_build_client())) + self._retrieval = V2RagRetrievalAdapter(retriever) + self._evidence = DocsEvidenceAssembler() + self._gate = DocsEvidenceGate() + self._summary_graph = DocsExplainSummaryGraph(_build_v2_llm()) + self._find_files_graph = DocsExplainFindFilesGraph() + self._general_graph = GeneralSummaryGraph(_build_v2_llm()) + + def execute(self, case: V3Case, rag_session_id: str | None) -> ExecutionPayload: + return asyncio.run(self._execute_async(case, rag_session_id)) + + async def _execute_async(self, case: V3Case, rag_session_id: str | None) -> ExecutionPayload: + runtime = _RuntimeStub(query=case.query) + route = self._router.route(case.query) + _log_pipeline_step( + runtime, + "router_resolved", + { + "domain": route.routing_domain, + "intent": route.intent, + "subintent": route.subintent, + "confidence": route.confidence, + }, + ) + _log_pipeline_step( + runtime, + "anchors_extracted", + { + "signal_types": route_anchor_summary(route)["signal_types"], + "endpoint_paths": route.anchors.endpoint_paths, + "target_doc_hints": route.anchors.target_doc_hints, + "matched_aliases": route.anchors.matched_aliases, + }, + ) + _log_pipeline_step( + runtime, + "alias_resolution", + { + "resolved_aliases": route.anchors.matched_aliases, + "target_doc_hints": route.anchors.target_doc_hints, + }, + ) + if case.mode == "router_only": + return ExecutionPayload( + actual=_actual_from_v2(route), + details=_details(case.query, route=route, pipeline_steps=_build_pipeline_steps(runtime.logs)), + ) + plan = self._policy.resolve(route) + _log_pipeline_step( + runtime, + "retrieval_profile_selected", + {"profile": plan.profile, "layers": plan.layers, "filters": plan.filters}, + ) + semantic_rows = await self._retrieve_rows(route, rag_session_id, plan) + seeded_rows = await self._seed_candidates_from_target_hints(route, rag_session_id, plan) + metadata_rows = self._metadata_lookup_candidates([*seeded_rows, *semantic_rows], route) + rows = self._merge_candidate_rows(seeded_rows, metadata_rows, semantic_rows) + _log_pipeline_step( + runtime, + "candidate_generation", + { + "resolved_aliases": route.anchors.matched_aliases, + "target_doc_hints": route.anchors.target_doc_hints, + "candidate_docs_before_ranking": [self._trace_row(row) for row in rows[:8]], + "sources": { + "seeded": [self._trace_row(row) for row in seeded_rows[:5]], + "metadata_lookup": [self._trace_row(row) for row in metadata_rows[:5]], + "semantic": [self._trace_row(row) for row in semantic_rows[:5]], + }, + }, + ) + _log_pipeline_step( + runtime, + "retrieval_executed", + { + "query": case.query, + "profile": plan.profile, + "row_count": len(rows), + "target_doc_hints": route.anchors.target_doc_hints, + "top_results": [self._trace_row(row) for row in rows[:5]], + }, + ) + if case.mode == "router_rag": + return ExecutionPayload( + actual=_actual_from_v2(route, rows=rows, plan=plan, answer_mode="partial"), + details=_details(case.query, route=route, plan=plan, rows=rows, pipeline_steps=_build_pipeline_steps(runtime.logs)), + ) + answer, evidence, gate = await self._run_workflow(runtime, route, rag_session_id, rows) + answer_mode = gate.answer_mode + _log_pipeline_step( + runtime, + "answer_generated", + {"answer_mode": answer_mode, "answer_length": len(answer)}, + ) + return ExecutionPayload( + actual=_actual_from_v2(route, rows=rows, plan=plan, answer=answer, answer_mode=answer_mode), + details=_details( + case.query, + route=route, + plan=plan, + rows=rows, + evidence=evidence, + answer=answer, + logs=runtime.logs, + pipeline_steps=_build_pipeline_steps(runtime.logs), + ), + ) + + async def _retrieve_rows(self, route, rag_session_id: str | None, plan) -> list[dict]: + if not rag_session_id: + if route.intent == V2Intent.GENERAL_QA: + return [] + raise ValueError("process_v2 cases with DOCS intent require rag_session_id") + return await self._retrieval.fetch_rows(rag_session_id, route.normalized_query, plan) + + async def _seed_candidates_from_target_hints(self, route, rag_session_id: str | None, plan) -> list[dict]: + if not rag_session_id or not route.anchors.target_doc_hints: + return [] + return await self._retrieval.fetch_exact_paths(rag_session_id, paths=route.anchors.target_doc_hints, layers=plan.layers) + + def _metadata_lookup_candidates(self, rows: list[dict], route) -> list[dict]: + return DocsMetadataLookupIndex(rows).lookup(route) + + def _merge_candidate_rows(self, *groups: list[dict]) -> list[dict]: + merged: list[dict] = [] + seen: set[tuple[str, str, str]] = set() + for rows in groups: + for row in rows: + key = ( + str(row.get("path") or ""), + str(row.get("layer") or ""), + str(dict(row.get("metadata") or {}).get("section_path") or ""), + ) + if key in seen: + continue + seen.add(key) + merged.append(row) + return merged + + async def _run_workflow( + self, + runtime: "_RuntimeStub", + route, + rag_session_id: str | None, + rows: list[dict], + ) -> tuple[str, dict, object]: + if route.intent == V2Intent.GENERAL_QA: + documents = self._evidence.assemble_summaries(rows, route) + gate = self._gate.check_summaries(route, documents) + _log_pipeline_step( + runtime, + "evidence_assembled", + {"mode": "summary", "primary_doc": documents[0].path if documents else None, "document_count": len(documents)}, + ) + self._log_ranking(runtime, documents) + _log_pipeline_step( + runtime, + "evidence_gate_checked", + {"passed": gate.passed, "reason": gate.reason, "answer_mode": gate.answer_mode}, + ) + context = GeneralSummaryContext(runtime=runtime, route=route, prompt_name="v2_general.summary_answer") + context.workflow_llm_enabled = self._workflow_llm_enabled + context.documents = documents + context.gate_decision = gate + final = await self._general_graph.run(context) + return final.answer, {"documents": [_serialize_summary(item) for item in documents], "files": []}, gate + if route.subintent == V2Subintent.FIND_FILES: + files = self._evidence.assemble_files(rows, route) + gate = self._gate.check_files(route, files) + _log_pipeline_step( + runtime, + "evidence_assembled", + {"mode": "find_files", "primary_file": files[0].path if files else None, "file_count": len(files)}, + ) + self._log_ranking(runtime, files) + _log_pipeline_step( + runtime, + "evidence_gate_checked", + {"passed": gate.passed, "reason": gate.reason, "answer_mode": gate.answer_mode}, + ) + context = DocsExplainFindFilesContext( + runtime=runtime, + route=route, + rag_session_id=rag_session_id or "", + files=files, + gate_decision=gate, + ) + final = await self._find_files_graph.run(context) + return final.answer, {"documents": [], "files": [_serialize_file(item) for item in files]}, gate + documents = self._evidence.assemble_summaries(rows, route) + gate = self._gate.check_summaries(route, documents) + _log_pipeline_step( + runtime, + "evidence_assembled", + {"mode": "summary", "primary_doc": documents[0].path if documents else None, "document_count": len(documents)}, + ) + self._log_ranking(runtime, documents) + _log_pipeline_step( + runtime, + "evidence_gate_checked", + {"passed": gate.passed, "reason": gate.reason, "answer_mode": gate.answer_mode}, + ) + context = DocsExplainSummaryContext( + runtime=runtime, + route=route, + rag_session_id=rag_session_id or "", + prompt_name="v2_docs_explain.summary_answer", + workflow_llm_enabled=self._workflow_llm_enabled, + documents=documents, + gate_decision=gate, + ) + final = await self._summary_graph.run(context) + return final.answer, {"documents": [_serialize_summary(item) for item in documents], "files": []}, gate + + def _trace_row(self, row: dict) -> dict[str, object]: + metadata = dict(row.get("metadata") or {}) + return { + "path": str(row.get("path") or ""), + "layer": str(row.get("layer") or ""), + "title": str(row.get("title") or ""), + "document_id": str(metadata.get("document_id") or metadata.get("doc_id") or ""), + } + + def _log_ranking(self, runtime: "_RuntimeStub", items: list) -> None: + top_docs: list[dict[str, object]] = [] + for item in items[:4]: + top_docs.append( + { + "doc": getattr(item, "path", ""), + "score": getattr(item, "score", 0), + "match_reason": getattr(item, "match_reason", ""), + } + ) + _log_pipeline_step( + runtime, + "ranking_explained", + { + "doc": getattr(item, "path", ""), + "score": getattr(item, "score", 0), + "score_breakdown": getattr(item, "score_breakdown", {}), + "match_reason": getattr(item, "match_reason", ""), + }, + ) + _log_pipeline_step( + runtime, + "ranking_explained", + { + "top_docs_after_ranking": top_docs, + "ranking_score_breakdown": [ + { + "doc": getattr(item, "path", ""), + "score_breakdown": getattr(item, "score_breakdown", {}), + } + for item in items[:4] + ], + }, + ) + + +@dataclass(slots=True) +class _RequestStub: + request_id: str + message: str + + +@dataclass(slots=True) +class _SessionStub: + active_rag_session_id: str | None = None + + +class _PublisherStub: + async def publish_status(self, request_id: str, source: str, message: str, payload: dict | None = None) -> None: + return None + + +class _TraceLoggerStub: + def __init__(self, store: list[dict]) -> None: + self._store = store + + def log_module(self, request_id: str, module: str, title: str, payload: dict | None = None) -> None: + self._store.append( + {"request_id": request_id, "module": module, "event": title, "payload": dict(payload or {})} + ) + + +class _RuntimeStub: + def __init__(self, *, query: str) -> None: + self.logs: list[dict] = [] + self.request = _RequestStub(request_id="pipeline_setup_v3", message=query) + self.session = _SessionStub() + self.publisher = _PublisherStub() + self.trace = RequestTraceContext(request_id=self.request.request_id, logger=_TraceLoggerStub(self.logs)) + + +def _build_client() -> GigaChatClient: + settings = GigaChatSettings.from_env() + return GigaChatClient(settings, GigaChatTokenProvider(settings)) + + +def _build_v2_llm() -> AgentLlmService: + prompt_paths = [ + Path(__file__).resolve().parents[3] / "src/app/core/agent/processes/v2/prompts.yml", + Path(__file__).resolve().parents[3] / "src/app/core/agent/processes/v2/general_prompts.yml", + Path(__file__).resolve().parents[3] / "src/app/core/agent/processes/v2/intent_router/routers/prompts.yml", + ] + return AgentLlmService(client=_build_client(), prompts=PromptLoader(prompt_paths)) + + +def _actual_from_v2(route, *, rows: list[dict] | None = None, plan=None, answer: str = "", answer_mode: str = "partial") -> dict: + return { + "domain": route.routing_domain, + "intent": route.intent, + "sub_intent": route.subintent, + "rag_count": len(rows or []), + "llm_answer": answer, + "answer_mode": answer_mode, + "path_scope": tuple(), + "symbol_candidates": tuple(), + "entity_candidates": tuple(_entity_candidates(rows or [])), + "doc_scope": tuple(_doc_scope(rows or [])), + "layers": tuple(getattr(plan, "layers", []) or []), + "filters": dict(getattr(plan, "filters", {}) or {}), + } + + +def _details(query: str, **payload) -> dict: + details = {"query": query} + for key, value in payload.items(): + if key == "route": + details["router_result"] = asdict(value) + elif key == "plan": + details["retrieval_plan"] = asdict(value) + else: + details[key] = value + return details + + +def _doc_scope(rows: list[dict]) -> list[str]: + values: list[str] = [] + for row in rows: + metadata = dict(row.get("metadata") or {}) + for candidate in (metadata.get("document_id"), metadata.get("doc_id"), row.get("path")): + value = str(candidate or "").strip() + if value and value not in values: + values.append(value) + return values + + +def _entity_candidates(rows: list[dict]) -> list[str]: + values: list[str] = [] + for row in rows: + metadata = dict(row.get("metadata") or {}) + value = str(metadata.get("entity_name") or row.get("title") or "").strip() + if value and value not in values and str(row.get("layer") or "") == "D3_ENTITY_CATALOG": + values.append(value) + return values + + +def _serialize_summary(item: RetrievedSummary) -> dict: + return asdict(item) + + +def _serialize_file(item: RetrievedFile) -> dict: + return asdict(item) + + +def _build_pipeline_steps(logs: list[dict]) -> list[dict]: + steps: list[dict] = [] + for item in logs: + if item.get("module") != "process.v2.pipeline": + continue + steps.append({"step": item.get("event"), "output": item.get("payload") or {}}) + return steps + + +def _log_pipeline_step(runtime: _RuntimeStub, step: str, payload: dict[str, object]) -> None: + runtime.logs.append( + { + "request_id": runtime.request.request_id, + "module": "process.v2.pipeline", + "event": step, + "payload": payload, + } + ) diff --git a/tests/pipeline_setup_v3/shared/env_loader.py b/tests/pipeline_setup_v3/shared/env_loader.py index cc289f9..36b45c0 100644 --- a/tests/pipeline_setup_v3/shared/env_loader.py +++ b/tests/pipeline_setup_v3/shared/env_loader.py @@ -3,7 +3,7 @@ from __future__ import annotations import os from pathlib import Path -from app.modules.shared.env_loader import load_workspace_env +from app.core.shared.config import load_workspace_env def load_pipeline_setup_env(start_dir: str | Path | None = None) -> list[Path]: diff --git a/tests/pipeline_setup_v3/shared/rag_indexer.py b/tests/pipeline_setup_v3/shared/rag_indexer.py index f1f4712..6a0d104 100644 --- a/tests/pipeline_setup_v3/shared/rag_indexer.py +++ b/tests/pipeline_setup_v3/shared/rag_indexer.py @@ -8,7 +8,7 @@ from typing import TYPE_CHECKING from uuid import uuid4 if TYPE_CHECKING: - from app.modules.rag.persistence.repository import RagRepository + from app.core.rag.persistence.repository import RagRepository class LocalRepoFileCollector: @@ -77,7 +77,7 @@ class DeterministicEmbedder: class RagSessionIndexer: def __init__(self, repository: "RagRepository") -> None: - from app.modules.rag.services.rag_service import RagService + from app.core.rag.indexing.service import RagService self._repository = repository self._rag = RagService(embedder=DeterministicEmbedder(), repository=repository) @@ -90,7 +90,7 @@ class RagSessionIndexer: files = LocalRepoFileCollector(repo_path).collect() if not files: raise ValueError(f"No indexable text files found under: {repo_path}") - logger = logging.getLogger("app.modules.rag.services.rag_service") + logger = logging.getLogger("app.core.rag.indexing.service") previous_level = logger.level logger.setLevel(logging.ERROR) try: diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..a32f7e3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,43 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "Информация о том, как именно работает `/health`, не представлена в предоставленных SUMMARY-блоках. \n\nФайлы, на которые я опирался:\n- docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..23cb454 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,51 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "Информация о том, как именно работает `/health`, не представлена в предоставленных SUMMARY-блоках. \n\nФайлы, на которые я опирался:\n- docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..7f77475 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,43 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'message'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "На основе представленного SUMMARY-блока, документация о том, что именно делает endpoint `/send`, отсутствует. В контексте указано, что он связан с отправкой сообщений, но подробностей его работы не приведено.\n\nФайлы, на которые опирался ответ:\n- docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..4338230 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,51 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "На основе представленного SUMMARY-блока, документация о том, что именно делает endpoint `/send`, отсутствует. В контексте указано, что он связан с отправкой сообщений, но подробностей его работы не приведено.\n\nФайлы, на которые опирался ответ:\n- docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'message' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..5940c38 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,30 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "retrieval: expected non-empty rag, got 0 rows", + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got []" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "`/actions/{action}` обычно используется в API для выполнения конкретных действий или операций, где `{action}` заменяется названием нужного действия. Это позволяет организовывать и вызывать различные функции через единый интерфейс.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..654ade3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,38 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "`/actions/{action}` обычно используется в API для выполнения конкретных действий или операций, где `{action}` заменяется названием нужного действия. Это позволяет организовывать и вызывать различные функции через единый интерфейс.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- retrieval: expected non-empty rag, got 0 rows +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got [] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..d367b96 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,30 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "retrieval: expected non-empty rag, got 0 rows", + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got []" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Runtime health — это процесс мониторинга и обеспечения стабильности работы системы или приложения во время их выполнения (runtime), позволяющий своевременно обнаруживать и устранять возникающие проблемы.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..b834bc1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,38 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Runtime health — это процесс мониторинга и обеспечения стабильности работы системы или приложения во время их выполнения (runtime), позволяющий своевременно обнаруживать и устранять возникающие проблемы.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- retrieval: expected non-empty rag, got 0 rows +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got [] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..554b88a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,33 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "retrieval: expected non-empty rag, got 0 rows", + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got []", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Цикл отправки уведомлений включает следующие шаги: отлавливание события, обработка данных, выбор получателей, формирование сообщения и его отправка выбранным пользователям.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..b600c8e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,41 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Цикл отправки уведомлений включает следующие шаги: отлавливание события, обработка данных, выбор получателей, формирование сообщения и его отправка выбранным пользователям.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- retrieval: expected non-empty rag, got 0 rows +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got [] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..708a5c2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,32 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "retrieval: expected non-empty rag, got 0 rows", + "doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got []", + "llm: expected answer to contain 'RuntimeManager'", + "llm: expected answer to contain 'TelegramControlChannel'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Архитектура Telegram Notify App основана на клиент-серверной модели, где серверное приложение обрабатывает входящие данные и отправляет уведомления через Telegram API. Клиентское приложение взаимодействует с сервером для отправки уведомлений. Пользователь настраивает параметры уведомлений в приложении, а сервер передает их в виде сообщений в указанный Telegram-канал или пользователю.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..101ae89 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,40 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Архитектура Telegram Notify App основана на клиент-серверной модели, где серверное приложение обрабатывает входящие данные и отправляет уведомления через Telegram API. Клиентское приложение взаимодействует с сервером для отправки уведомлений. Пользователь настраивает параметры уведомлений в приложении, а сервер передает их в виде сообщений в указанный Telegram-канал или пользователю.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- retrieval: expected non-empty rag, got 0 rows +- doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got [] +- llm: expected answer to contain 'RuntimeManager' +- llm: expected answer to contain 'TelegramControlChannel' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..e9fe557 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,32 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "retrieval: expected non-empty rag, got 0 rows", + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got []", + "doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got []", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Worker в этом приложении выполняет фоновые задачи или процессы, позволяя основному приложению работать быстро и плавно. Это может включать обработку данных, отправку уведомлений, загрузку файлов и другие ресурсоемкие операции.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..3851457 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,40 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Worker в этом приложении выполняет фоновые задачи или процессы, позволяя основному приложению работать быстро и плавно. Это может включать обработку данных, отправку уведомлений, загрузку файлов и другие ресурсоемкие операции.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- retrieval: expected non-empty rag, got 0 rows +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got [] +- doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got [] +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..13f4365 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,47 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "На основе представленного SUMMARY-блока конкретной информации о статусах здоровья runtime нет. Файл, связанный с доменом \"runtime health\" (../domains/runtime-health-entity.md), вероятно, содержит эту информацию, но она не включена в текущий контекст.\n\nФайлы, на которые опирался ответ:\n- docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..7014fc0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,55 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "На основе представленного SUMMARY-блока конкретной информации о статусах здоровья runtime нет. Файл, связанный с доменом \"runtime health\" (../domains/runtime-health-entity.md), вероятно, содержит эту информацию, но она не включена в текущий контекст.\n\nФайлы, на которые опирался ответ:\n- docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..460127a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,33 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "retrieval: expected non-empty rag, got 0 rows", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got []", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Ручная отправка сообщения в приложении обычно осуществляется нажатием кнопки \"Отправить\" или аналогичного элемента интерфейса после того, как пользователь введет текст сообщения.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..d843338 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,41 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Ручная отправка сообщения в приложении обычно осуществляется нажатием кнопки \"Отправить\" или аналогичного элемента интерфейса после того, как пользователь введет текст сообщения.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- retrieval: expected non-empty rag, got 0 rows +- doc_scope should contain 'docs/api/send-message-endpoint.md', got [] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..f9f85df --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,34 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "retrieval: expected non-empty rag, got 0 rows", + "retrieval: expected at least 2 rows, got 0", + "doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got []", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got []", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain '/send'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Сервис может отправлять и получать сообщения, реагировать на команды и запросы пользователей через Telegram-бота. Взаимодействие происходит посредством отправки запросов и получения ответов от бота в чате.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..229fb96 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,42 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Сервис может отправлять и получать сообщения, реагировать на команды и запросы пользователей через Telegram-бота. Взаимодействие происходит посредством отправки запросов и получения ответов от бота в чате.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- retrieval: expected non-empty rag, got 0 rows +- retrieval: expected at least 2 rows, got 0 +- doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got [] +- doc_scope should contain 'docs/api/send-message-endpoint.md', got [] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain '/send' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..b951146 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,42 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..28203f3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,50 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..a7d3c2d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,42 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..263c1b6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,50 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..d4a94dc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,42 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..e0b6625 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,50 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..2fa11f9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,42 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/architecture/telegram-notify-app-overview.md`\n- `docs/README.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..f1d4543 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,50 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/architecture/telegram-notify-app-overview.md`\n- `docs/README.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..3c55d0f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,42 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "RuntimeManager", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..51f00e3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,50 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "RuntimeManager", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..27afb1d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,40 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..12e65d4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,50 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..c5c3b87 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,42 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..44eb963 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,50 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..4246968 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,40 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager", + "TelegramSendService" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..832f39b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,50 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager", + "TelegramSendService" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..e52b292 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,43 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..5b527b0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,51 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/README.md`\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..7169076 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,43 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..c674b3a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,51 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 20, + "llm_answer": "Нашёл релевантные файлы документации:\n- `docs/architecture/telegram-notify-app-overview.md`", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D3_ENTITY_CATALOG", + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..b718226 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,27 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Это зависит от контекста. Уточни, пожалуйста, о каком именно сервисе идет речь.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..1d5f9f6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,35 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Это зависит от контекста. Уточни, пожалуйста, о каком именно сервисе идет речь.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..81dee19 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,25 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Test_echo_app нужен для тестирования и отладки приложений и систем, повторяя входящий запрос в качестве ответа, чтобы убедиться в правильной работе связи и функциональности.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..ed9376c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,35 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Test_echo_app нужен для тестирования и отладки приложений и систем, повторяя входящий запрос в качестве ответа, чтобы убедиться в правильной работе связи и функциональности.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..4664e67 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,27 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Приложение решает задачу (указать конкретную задачу, например, управления контактами, планирования задач, учета финансов и т.д.) для пользователей путем предоставления удобного и интуитивно понятного интерфейса.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..fc0d6ea --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,35 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Приложение решает задачу (указать конкретную задачу, например, управления контактами, планирования задач, учета финансов и т.д.) для пользователей путем предоставления удобного и интуитивно понятного интерфейса.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..5615b14 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,43 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "Документация проекта включает следующие компоненты:\n\n- Архитектура приложения `test_echo_app`\n- Описание HTTP API (control plane)\n- Цикл отправки уведомлений\n- Health-модель\n- Каталог ошибок\n\nСтруктура документации канонически разделена на несколько секций: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`. Основная родительская документация — \"Архитектура Telegram Notify App\" (находится по пути ./architecture/telegram-notify-app-overview.md). Документы взаимосвязаны с помощью ссылок `related_docs`, `parent`/`children` и обычных markdown-ссылок без повторения одной и той же информации.\n\nФайлы, использованные для ответа: docs/README.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..226c40b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,51 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "Документация проекта включает следующие компоненты:\n\n- Архитектура приложения `test_echo_app`\n- Описание HTTP API (control plane)\n- Цикл отправки уведомлений\n- Health-модель\n- Каталог ошибок\n\nСтруктура документации канонически разделена на несколько секций: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`. Основная родительская документация — \"Архитектура Telegram Notify App\" (находится по пути ./architecture/telegram-notify-app-overview.md). Документы взаимосвязаны с помощью ссылок `related_docs`, `parent`/`children` и обычных markdown-ссылок без повторения одной и той же информации.\n\nФайлы, использованные для ответа: docs/README.md", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..03d8590 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,27 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Основные части любой системы включают вход, процессоры (обработчики), память и выход.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..e83b3a0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,35 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Основные части любой системы включают вход, процессоры (обработчики), память и выход.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..be43495 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,25 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Приложение обычно состоит из пользовательского интерфейса (то, что видит и с чем взаимодействует пользователь), логики приложения (код, который обрабатывает действия пользователя) и базы данных (место хранения информации). Все эти части связаны между собой для обеспечения удобной работы пользователя.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..7c7b4b5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,35 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Приложение обычно состоит из пользовательского интерфейса (то, что видит и с чем взаимодействует пользователь), логики приложения (код, который обрабатывает действия пользователя) и базы данных (место хранения информации). Все эти части связаны между собой для обеспечения удобной работы пользователя.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..66911fe --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,25 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Кроме API, здесь есть дружелюбная атмосфера, готовность помочь и интересные беседы на различные темы.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..f1f0cf8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,35 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Кроме API, здесь есть дружелюбная атмосфера, готовность помочь и интересные беседы на различные темы.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..3548b3f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,25 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Telegram Notify App состоит из простого пользовательского интерфейса для отправки уведомлений, механизма обработки сообщений и интеграции с API Telegram, позволяющей отправлять сообщения в выбранные чаты или каналы.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..2e5b1c0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,35 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Telegram Notify App состоит из простого пользовательского интерфейса для отправки уведомлений, механизма обработки сообщений и интеграции с API Telegram, позволяющей отправлять сообщения в выбранные чаты или каналы.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..6e5d7fe --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,45 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "llm: expected answer to contain 'README'", + "llm: expected answer to contain 'архитект'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "Не нашёл подходящих SUMMARY-блоков в документации по этому запросу.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..2090b54 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,53 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 12, + "llm_answer": "Не нашёл подходящих SUMMARY-блоков в документации по этому запросу.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- llm: expected answer to contain 'README' +- llm: expected answer to contain 'архитект' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..7c84098 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,25 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Проектная документация включает описание целей, задач, методов, графика и бюджета проекта, а также список необходимых ресурсов и команду исполнителей.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..d55ff45 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,35 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "Проектная документация включает описание целей, задач, методов, графика и бюджета проекта, а также список необходимых ресурсов и команду исполнителей.", + "answer_mode": "answered", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +- none + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/summary.md new file mode 100644 index 0000000..871b698 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_194106/summary.md @@ -0,0 +1,153 @@ +# pipeline_setup_v3 summary + +Passed: 7/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-01-what-is-service**: llm: expected answer to contain 'Telegram' +- **v2-general-03-problem**: llm: expected answer to contain 'Telegram' +- **v2-general-04-docs-contents**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN +- **v2-general-05-main-parts**: llm: expected answer to contain 'Telegram' +- **v2-general-09-reading-order**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; llm: expected answer to contain 'README'; llm: expected answer to contain 'архитект' +- **v2-docs-summary-01-health**: llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'message' +- **v2-docs-summary-03-actions**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; retrieval: expected non-empty rag, got 0 rows; doc_scope should contain 'docs/api/control-actions-endpoint.md', got [] +- **v2-docs-summary-04-runtime-health**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; retrieval: expected non-empty rag, got 0 rows; doc_scope should contain 'docs/domains/runtime-health-entity.md', got [] +- **v2-docs-summary-05-notification-loop**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; retrieval: expected non-empty rag, got 0 rows; doc_scope should contain 'docs/logic/telegram-notification-loop.md', got []; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-06-architecture**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; retrieval: expected non-empty rag, got 0 rows; doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got []; llm: expected answer to contain 'RuntimeManager'; llm: expected answer to contain 'TelegramControlChannel' +- **v2-docs-summary-07-worker**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; retrieval: expected non-empty rag, got 0 rows; doc_scope should contain 'docs/logic/telegram-notification-loop.md', got []; doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got []; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; retrieval: expected non-empty rag, got 0 rows; doc_scope should contain 'docs/api/send-message-endpoint.md', got []; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; retrieval: expected non-empty rag, got 0 rows; retrieval: expected at least 2 rows, got 0; doc_scope should contain 'docs/architecture/telegram-notify-app-overview.md', got []; doc_scope should contain 'docs/api/send-message-endpoint.md', got []; llm: expected answer to contain 'worker'; llm: expected answer to contain '/send' +- **v2-find-files-01-health**: llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-07-errors-catalog**: llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-09-telegram-bot-api**: llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > Это зависит от контекста. Уточни, пожалуйста, о каком именно сервисе идет речь. +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > Test\_echo\_app нужен для тестирования и отладки приложений и систем, повторяя входящий запрос в качестве ответа, чтобы убедиться в правильной работе связи и функциональности. +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > Приложение решает задачу \(указать конкретную задачу, например, управления контактами, планирования задач, учета финансов и т.д.\) для пользователей путем предоставления удобного и интуитивно понятного интерфейса. +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > Документация проекта включает следующие компоненты: \- Архитектура приложения \`test\_echo\_app\` \- Описание HTTP API \(control plane\) \- Цикл отправки уведомлений \- Health\-модель \- Каталог ошибок Структура документации канонически разделена на несколько секций: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. Основная родительская документация — "Архитектура Telegram Notify App" \(находится по пути ./architecture/telegram\-notify\-app\-overview.md\). Документы взаимосвязаны с помощью ссылок \`related\_docs\`, \`parent\`/\`children\` и обычных markdown\-ссылок без повторения одной и той же информации. Файлы, использованные для ответа: docs/README.md +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > Основные части любой системы включают вход, процессоры \(обработчики\), память и выход. +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > Приложение обычно состоит из пользовательского интерфейса \(то, что видит и с чем взаимодействует пользователь\), логики приложения \(код, который обрабатывает действия пользователя\) и базы данных \(место хранения информации\). Все эти части связаны между собой для обеспечения удобной работы пользователя. +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > Кроме API, здесь есть дружелюбная атмосфера, готовность помочь и интересные беседы на различные темы. +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > Telegram Notify App состоит из простого пользовательского интерфейса для отправки уведомлений, механизма обработки сообщений и интеграции с API Telegram, позволяющей отправлять сообщения в выбранные чаты или каналы. +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > Не нашёл подходящих SUMMARY\-блоков в документации по этому запросу. +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > Проектная документация включает описание целей, задач, методов, графика и бюджета проекта, а также список необходимых ресурсов и команду исполнителей. +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > Информация о том, как именно работает \`/health\`, не представлена в предоставленных SUMMARY\-блоках. Файлы, на которые я опирался: \- docs/architecture/telegram\-notify\-app\-overview.md +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > На основе представленного SUMMARY\-блока, документация о том, что именно делает endpoint \`/send\`, отсутствует. В контексте указано, что он связан с отправкой сообщений, но подробностей его работы не приведено. Файлы, на которые опирался ответ: \- docs/architecture/telegram\-notify\-app\-overview.md +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > \`/actions/\{action\}\` обычно используется в API для выполнения конкретных действий или операций, где \`\{action\}\` заменяется названием нужного действия. Это позволяет организовывать и вызывать различные функции через единый интерфейс. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > Runtime health — это процесс мониторинга и обеспечения стабильности работы системы или приложения во время их выполнения \(runtime\), позволяющий своевременно обнаруживать и устранять возникающие проблемы. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > Цикл отправки уведомлений включает следующие шаги: отлавливание события, обработка данных, выбор получателей, формирование сообщения и его отправка выбранным пользователям. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > Архитектура Telegram Notify App основана на клиент\-серверной модели, где серверное приложение обрабатывает входящие данные и отправляет уведомления через Telegram API. Клиентское приложение взаимодействует с сервером для отправки уведомлений. Пользователь настраивает параметры уведомлений в приложении, а сервер передает их в виде сообщений в указанный Telegram\-канал или пользователю. +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > Worker в этом приложении выполняет фоновые задачи или процессы, позволяя основному приложению работать быстро и плавно. Это может включать обработку данных, отправку уведомлений, загрузку файлов и другие ресурсоемкие операции. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > На основе представленного SUMMARY\-блока конкретной информации о статусах здоровья runtime нет. Файл, связанный с доменом "runtime health" \(../domains/runtime\-health\-entity.md\), вероятно, содержит эту информацию, но она не включена в текущий контекст. Файлы, на которые опирался ответ: \- docs/architecture/telegram\-notify\-app\-overview.md +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > Ручная отправка сообщения в приложении обычно осуществляется нажатием кнопки "Отправить" или аналогичного элемента интерфейса после того, как пользователь введет текст сообщения. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > Сервис может отправлять и получать сообщения, реагировать на команды и запросы пользователей через Telegram\-бота. Взаимодействие происходит посредством отправки запросов и получения ответов от бота в чате. +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > Нашёл релевантные файлы документации: \- \`docs/README.md\` \- \`docs/architecture/telegram\-notify\-app\-overview.md\` +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > Нашёл релевантные файлы документации: \- \`docs/README.md\` \- \`docs/architecture/telegram\-notify\-app\-overview.md\` +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > Нашёл релевантные файлы документации: \- \`docs/README.md\` \- \`docs/architecture/telegram\-notify\-app\-overview.md\` +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > Нашёл релевантные файлы документации: \- \`docs/architecture/telegram\-notify\-app\-overview.md\` \- \`docs/README.md\` +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > Нашёл релевантные файлы документации: \- \`docs/README.md\` \- \`docs/architecture/telegram\-notify\-app\-overview.md\` +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > Нашёл релевантные файлы документации: \- \`docs/README.md\` \- \`docs/architecture/telegram\-notify\-app\-overview.md\` +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > Нашёл релевантные файлы документации: \- \`docs/README.md\` +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > Нашёл релевантные файлы документации: \- \`docs/README.md\` +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > Нашёл релевантные файлы документации: \- \`docs/README.md\` \- \`docs/architecture/telegram\-notify\-app\-overview.md\` +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > Нашёл релевантные файлы документации: \- \`docs/architecture/telegram\-notify\-app\-overview.md\` \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..f750d75 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,153 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..72de4ec --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,186 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..1c3c7bf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,152 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..010899d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,185 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..f91e458 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,153 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..677f559 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,186 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..15376d0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,139 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_domain_entity", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..e8a24ab --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,172 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..c159648 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,143 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_logic_flow", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..b5fee52 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,176 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..24b90d1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,124 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_architecture", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..7a22ca8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,159 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..87eef02 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,129 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_logic_flow", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..1e5bd47 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,162 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..0f7c40a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,143 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_domain_entity", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..8afa3cb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,176 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..11bf82f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,151 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramNotifyModule", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..851ead9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,184 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyWorker", + "RuntimeManager", + "TelegramNotifyModule", + "TelegramControlChannel" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..a185ba1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,127 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..8fa8c48 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,160 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..13b57c3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,137 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..a31fc92 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,170 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..479930f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,137 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..03887d8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,170 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService", + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..57d42de --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,139 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..91dd829 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,172 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..14d17fd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,141 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "sub_intent: expected FIND_FILES, got SUMMARY", + "anchor_type: expected FIND_FILES, got DOMAIN_ENTITY", + "answer_mode: expected deterministic, got insufficient_evidence", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_domain_entity", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..dbc9ebb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,174 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 8, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- sub_intent: expected FIND_FILES, got SUMMARY +- anchor_type: expected FIND_FILES, got DOMAIN_ENTITY +- answer_mode: expected deterministic, got insufficient_evidence +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..56ac770 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,125 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "RuntimeManager", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..04c174a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,158 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "RuntimeManager", + "TelegramControlChannel" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..c075007 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,134 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 1 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 49 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..0f8f3ab --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,167 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramSendService", + "TelegramNotifyModule", + "TelegramControlChannel", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 1 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 49 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..dfcb041 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,125 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..7afdd0b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,158 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..7e8aae0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,124 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager", + "TelegramSendService" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..42f891a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,157 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager", + "TelegramSendService" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..3954aae --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,126 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..01125b6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,159 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..fe24022 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,126 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..7a65427 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,159 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "row_count": 12, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..c39257c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,122 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..ae8cdd9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,157 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..64cabe6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,122 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..2825dc5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,157 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..89fa183 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,122 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..4fa7105 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,157 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..66f480a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,122 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..6e8808c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,157 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..5536124 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,124 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..d784f18 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,157 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..6542c3b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,124 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..dbe6f5b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,157 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..585af0c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,122 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..0288a32 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,157 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..6120517 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,129 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "docs_summary_architecture", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..1d26982 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,162 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramSendService", + "TelegramControlChannel", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "row_count": 8, + "top_files": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..de9d221 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,124 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..4cdbe42 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,157 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..20b7626 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,122 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..6f07a1a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,157 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "row_count": 8, + "top_files": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/summary.md new file mode 100644 index 0000000..a47c9df --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_201415/summary.md @@ -0,0 +1,153 @@ +# pipeline_setup_v3 summary + +Passed: 7/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- **v2-find-files-01-health**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: sub_intent: expected FIND_FILES, got SUMMARY; anchor_type: expected FIND_FILES, got DOMAIN_ENTITY; answer_mode: expected deterministic, got insufficient_evidence; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-06-architecture**: answer_mode: expected deterministic, got deterministic_file_list +- **v2-find-files-07-errors-catalog**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-08-docs-index**: answer_mode: expected deterministic, got deterministic_file_list +- **v2-find-files-09-telegram-bot-api**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md, docs/README.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..275d186 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,189 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..210e5d1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,226 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..973244c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,172 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..44c8469 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,205 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..20081c5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,189 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..72853a7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,226 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..3d93acc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,175 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..5b5e82d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,212 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..c35f318 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,196 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..82952e7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,237 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..d72cbe2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,195 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..070f32c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,238 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..148d944 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,175 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..1b985e8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,212 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..d007838 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,179 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..3ff017e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,216 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..b4f3f03 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,187 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..7e707c3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,224 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..9a43a17 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,188 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..01f40f6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,229 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..bcee94d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,196 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле описан `/health`?", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..6ae4e4d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,237 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле описан `/health`?", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..dfb8fe5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,196 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..e09680e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,237 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..906d1f7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,198 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..7793bbf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,239 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..7e5e8fd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,177 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "sub_intent: expected FIND_FILES, got SUMMARY", + "anchor_type: expected FIND_FILES, got DOMAIN_ENTITY", + "answer_mode: expected deterministic, got insufficient_evidence", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком документе описан runtime health?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..2e9fef0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,214 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком документе описан runtime health?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- sub_intent: expected FIND_FILES, got SUMMARY +- anchor_type: expected FIND_FILES, got DOMAIN_ENTITY +- answer_mode: expected deterministic, got insufficient_evidence +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..f844491 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,194 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..8ec030c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,235 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..277b986 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,176 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 15, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где описана архитектура приложения?", + "profile": "docs_find_files_catalog", + "row_count": 15, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 49 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..a251b50 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,213 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 15, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где описана архитектура приложения?", + "profile": "docs_find_files_catalog", + "row_count": 15, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 49 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..7573b67 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,186 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "docs_find_files_catalog", + "row_count": 12, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..d686b1a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,227 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager", + "TelegramControlChannel", + "TelegramNotifyModule", + "TelegramSendService", + "TelegramNotifyWorker" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "docs_find_files_catalog", + "row_count": 12, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..e8caa7d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,185 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager", + "TelegramSendService" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "docs_find_files_catalog", + "row_count": 12, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..1575d61 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,226 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramNotifyWorker", + "TelegramControlChannel", + "RuntimeManager", + "TelegramSendService" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "docs_find_files_catalog", + "row_count": 12, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..b91d7b4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,187 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "docs_find_files_catalog", + "row_count": 12, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..c7696c6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,228 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule", + "TelegramSendService", + "TelegramControlChannel", + "TelegramNotifyWorker", + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "docs_find_files_catalog", + "row_count": 12, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..1c96539 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,195 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected deterministic, got deterministic_file_list", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..6ec5ba4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,236 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 8, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic_file_list", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_find_files_catalog", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "docs_find_files_catalog", + "row_count": 8, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic_file_list" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic_file_list", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected deterministic, got deterministic_file_list +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..c0f2d12 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,183 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..2557e94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,226 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..0c35f52 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,183 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..5b2db3b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,226 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..8165769 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,183 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..c2c0dae --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,226 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..0a85bfa --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,183 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..a90db6c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,226 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..d70594f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,185 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..4be3f10 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,226 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..e08008d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,185 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..a94d31c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,226 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..5d1196a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,183 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..fd459c4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,226 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..dbbe189 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,198 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..4b3e4fe --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,239 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..fd086ce --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,185 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..7f9590a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,226 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..0a0505c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,183 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..a80bdf3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,226 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/summary.md new file mode 100644 index 0000000..0265d4c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_202730/summary.md @@ -0,0 +1,153 @@ +# pipeline_setup_v3 summary + +Passed: 7/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- **v2-find-files-01-health**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: sub_intent: expected FIND_FILES, got SUMMARY; anchor_type: expected FIND_FILES, got DOMAIN_ENTITY; answer_mode: expected deterministic, got insufficient_evidence; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-06-architecture**: answer_mode: expected deterministic, got deterministic_file_list +- **v2-find-files-07-errors-catalog**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-08-docs-index**: answer_mode: expected deterministic, got deterministic_file_list +- **v2-find-files-09-telegram-bot-api**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: answer_mode: expected deterministic, got deterministic_file_list; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md, docs/README.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..1c20004 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,295 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..b358832 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,344 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..75c76f5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,259 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..b31c739 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,304 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..6196dff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,299 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..65c69a1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,348 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..deb6ae2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,285 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..3ccb80d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,334 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..f23f2b3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,330 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..963fd2f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,383 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..7e842f1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,390 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..257f09f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,445 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..fefba8a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,281 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..36f453c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,330 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..0cdb688 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,289 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..b9ceaa5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,338 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..17ded9c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,297 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..94c1b16 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,346 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..1e0b5f0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,338 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..7e1df31 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,391 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..78d41ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,298 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..5207ae9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,351 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..863b971 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,298 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..396c5d9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,351 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..3be1e4a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,304 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..ce31457 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,357 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..9f88ec2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,302 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..e759ee7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,355 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..672c6d5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,296 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..e33bff1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,349 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..ba2f015 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..7fc62c3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,399 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..238cb6e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,302 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..348b8b0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,355 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..09e5785 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,314 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..08bb57d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,369 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..67047e6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,288 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..c4d79a7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,341 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..52da710 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,304 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..28d4704 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,357 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..0cf24da --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..302917b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,381 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..f047edf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..8e52fc6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,394 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..3583e60 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..475e3e0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,381 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..ad7cafc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..2de2e31 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,381 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..63a6a58 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..cf79327 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,381 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..46bf39b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..c2b9b4d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,381 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..750d6bb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..493a888 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,381 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..d2f8d32 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,389 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..11ee6e7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,442 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..194372a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..e9a5323 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,381 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..2545ba8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..7e55ef2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,381 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/summary.md new file mode 100644 index 0000000..632bd37 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_212743/summary.md @@ -0,0 +1,151 @@ +# pipeline_setup_v3 summary + +Passed: 9/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- **v2-find-files-01-health**: llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-07-errors-catalog**: llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-09-telegram-bot-api**: llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md, docs/README.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..785a289 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,306 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..79b08f7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,355 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..9ef2146 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,270 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..c882366 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,315 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..4bdcbdd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,310 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..1be8ecb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,359 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..c951544 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,296 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..2042605 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,345 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..f632b15 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,341 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..d70e3c5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,394 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..4f976de --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,416 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..db8eef6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,471 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..4296d69 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,292 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..73e8540 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,341 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..9fad7e4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,300 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..77f9c7c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,349 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..0973d8f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,308 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..fde410e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,357 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..65bc22d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,351 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..43a1035 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,404 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..99329b2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,308 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..ae827c8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,361 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..d444a74 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,308 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..71ef0b4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,361 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..22a26af --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,314 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..7daa0f1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,367 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..c509e60 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,312 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..79e0a20 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,365 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..726a2fc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,306 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..3ec8e9c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,359 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..01cb749 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,412 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 19, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..032440b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,467 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 19, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..a5feeeb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,312 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..e61ad85 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,365 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..f38ba3e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,399 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "docs/README.md", + "index.test_echo_app_docs", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/README.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 12, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..5b563e4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,454 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "docs/README.md", + "index.test_echo_app_docs", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/README.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 12, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..8f70615 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,296 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..1f7c1c1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,349 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..9506ba6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,314 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..b28dffc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,367 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..cefa3db --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..a325ea4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,394 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..0a60fbd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,352 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..aff2f61 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,407 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..94db832 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..93bdada --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,394 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..69746f9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..44ec97a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,394 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..84f10f3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,341 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..e98e7fe --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,394 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..831b0fc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,341 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..b6f0cde --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,394 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..1898d48 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..24c384d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,394 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..b0b464c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,415 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..c363018 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,468 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..f07f33b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,341 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..60fa796 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,394 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..e850984 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..1de0242 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,394 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/summary.md new file mode 100644 index 0000000..632bd37 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_215439/summary.md @@ -0,0 +1,151 @@ +# pipeline_setup_v3 summary + +Passed: 9/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- **v2-find-files-01-health**: llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-07-errors-catalog**: llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-09-telegram-bot-api**: llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md, docs/README.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..1042ef3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,311 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..793386f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,360 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..8022dce --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,275 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..1b91e04 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,320 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..75b1274 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,315 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..f83393e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,364 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..fc36a71 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,301 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..d197223 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,350 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..3ee99db --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,346 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 8 + } + }, + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..c73c55e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,399 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 8 + } + }, + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..a6cdc81 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,421 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 98, + "metadata_lookup": 2, + "semantic": 8 + } + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..62f3d50 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,476 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 98, + "metadata_lookup": 2, + "semantic": 8 + } + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..c26e843 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,297 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..01a2efd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,346 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..53550ad --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,305 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..4e552d4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,354 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..97ab490 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,313 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..f7de916 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,362 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..5123e8e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,356 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..db8e506 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,409 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..9b76f8f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,313 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..f96df8e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,366 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..d19f694 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,313 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..fba7504 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,366 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..d2336d8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,319 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..786907d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,372 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..a8e745b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,317 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..24f6838 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,370 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..98487fd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,311 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..2cca623 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,364 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..6e82a61 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,417 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 19, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 98, + "metadata_lookup": 2, + "semantic": 7 + } + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..37499d5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,472 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 19, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 98, + "metadata_lookup": 2, + "semantic": 7 + } + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..ca6dfd2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,317 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..4944887 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,370 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 7 + } + }, + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..615e745 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,404 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "docs/README.md", + "index.test_echo_app_docs", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/README.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 42, + "metadata_lookup": 1, + "semantic": 7 + } + }, + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 12, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..25d41fb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,459 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 12, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "docs/README.md", + "index.test_echo_app_docs", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/README.md" + ], + "candidates_before_ranking": [ + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 42, + "metadata_lookup": 1, + "semantic": 7 + } + }, + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 12, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..8fa580c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,301 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..d01a16f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,354 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..0c4ba6b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,319 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..d76060d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,372 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 1, + "semantic": 7 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..d96e69b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..f325154 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,399 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..5e3a597 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,357 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 2, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..61f9fff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,412 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 2, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..b1d6ef2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..e39e5f3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,399 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..1af8d41 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..96d9488 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,399 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..9d1deca --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,346 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..3af825e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,399 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..103eed5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,346 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..e9ffca4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,399 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..2c24326 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..099002f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,399 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..219e609 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,420 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 98, + "metadata_lookup": 2, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..3e83cb0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,473 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 19, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 98, + "metadata_lookup": 2, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D2_FACT_INDEX", + "title": "architecture.telegram_notify_app:mentions_entity", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 19, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..29876d1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,346 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..b49f906 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,399 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..dd8391a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..ac3e1ff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,399 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/architecture/telegram-notify-app-overview.md", + "docs/README.md", + "docs/README.md", + "docs/README.md" + ], + "candidate_sources": { + "seeded": 0, + "metadata_lookup": 0, + "semantic": 8 + } + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/summary.md new file mode 100644 index 0000000..632bd37 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260406_222632/summary.md @@ -0,0 +1,151 @@ +# pipeline_setup_v3 summary + +Passed: 9/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- **v2-find-files-01-health**: llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-07-errors-catalog**: llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-09-telegram-bot-api**: llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md, docs/README.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532.zip b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532.zip new file mode 100644 index 0000000..87e2090 Binary files /dev/null and b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532.zip differ diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..24e4724 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,278 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/readme.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..f6b826c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,327 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/readme.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..baa4326 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,242 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..15f051c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,287 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..ff05db4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,246 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..5b32579 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,291 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..5facb76 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,219 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 1, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 1, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..2ebbf68 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,264 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 1, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 1, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..11787e7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,279 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/readme.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..ed04fcb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,328 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/readme.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..192639a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,343 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 2, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/readme.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 2, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/readme.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..e9f44c1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,398 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 2, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/readme.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 2, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/readme.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..f088444 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,229 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..845b285 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,274 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..761b90f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,223 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 1, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 1, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..7a91f8b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,268 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 1, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 1, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..7465f31 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,244 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..c5941ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,289 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 2, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 2, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..8832505 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,259 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain '/send'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 2, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 503 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..d1c4cb5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,308 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 2, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 503 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain '/send' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..ec6c769 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,295 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..cabe220 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,348 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..55f48e7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,295 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/readme.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..710acd9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,348 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/readme.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..b79998a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,301 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + }, + { + "doc": "docs/readme.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..7d3aa54 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,354 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + }, + { + "doc": "docs/readme.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..6ac04b2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,299 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/readme.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..e0f251e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,352 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/readme.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..0af3f2c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,293 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..80c75cc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,346 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..3c6e75d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/readme.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..0d9949f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,394 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/readme.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..abd1d19 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,299 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..cdd064c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,352 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..6aa19d7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,334 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/README.md" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..53a4120 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,389 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/readme.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/README.md" + ], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D5_RELATION_GRAPH", + "title": "index.test_echo_app_docs:related_doc", + "document_id": "" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/readme.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..309f8a7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,283 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 710, + "score_breakdown": { + "semantic": 10, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 710, + "match_reason": "exact_title" + }, + { + "doc": "docs/readme.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..fb23481 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,336 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 710, + "score_breakdown": { + "semantic": 10, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 710, + "match_reason": "exact_title" + }, + { + "doc": "docs/readme.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..90acd95 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,301 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/readme.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..4fba286 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,354 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 2, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/readme.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 2, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/readme.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..16bad13 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,245 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 503 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..874d85f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,296 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 503 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..f6b0284 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,225 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain 'test_echo_app'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..081463c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,270 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain 'test_echo_app' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..70fa4ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,212 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..f12bdca --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,257 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..2cac079 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,211 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'архитект'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..5c46c9a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,256 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'архитект' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..524ef22 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,212 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..5bcd8c1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,257 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..01b4fa7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,211 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..6c45cb4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,256 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..4c8194f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,211 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'API'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..6f6cc1b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,256 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'API' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..ba6f84d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,342 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 2, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/readme.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 2, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/readme.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..979075f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,395 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 2, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/readme.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidates_before_ranking": [ + "docs/architecture/telegram-notify-app-overview.md", + "docs/readme.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 2, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/readme.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..f9668d6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,247 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/readme.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 503 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..a179771 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,296 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/readme.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/readme.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/readme.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/readme.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 503 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..d284154 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,209 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..eec09a2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,256 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 2, + "llm_answer": "В найденной документации нет достаточной опоры для общего summary по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "details": { + "target_doc_hints": [], + "candidates_before_ranking": [ + "docs/readme.md", + "docs/architecture/telegram-notify-app-overview.md" + ] + }, + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 2, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "general_docs_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 77 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/summary.md new file mode 100644 index 0000000..28d6caa --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_095532/summary.md @@ -0,0 +1,155 @@ +# pipeline_setup_v3 summary + +Passed: 5/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-02-purpose**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain 'test_echo_app' +- **v2-general-03-problem**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain 'Telegram' +- **v2-general-04-docs-contents**: llm: expected answer to contain 'архитект' +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем'; llm: expected answer to contain 'Telegram' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-07-besides-api**: llm: expected answer to contain 'API' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain '/send' +- **v2-find-files-01-health**: llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-07-errors-catalog**: llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-09-telegram-bot-api**: llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > В найденной документации нет достаточной опоры для общего summary по запросу. +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > В найденной документации нет достаточной опоры для общего summary по запросу. +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > В найденной документации нет достаточной опоры для общего summary по запросу. +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > В найденной документации нет достаточной опоры для общего summary по запросу. +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > В найденной документации нет достаточной опоры для общего summary по запросу. +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > В найденной документации нет достаточной опоры для общего summary по запросу. +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/readme.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > В найденной документации нет достаточной опоры для общего summary по запросу. +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/readme.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/readme.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/readme.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/readme.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/readme.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/readme.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > docs/architecture/telegram\-notify\-app\-overview.md docs/readme.md +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/readme.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md docs/readme.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/readme.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/readme.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/architecture/telegram\-notify\-app\-overview.md docs/readme.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/readme.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..1c20004 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,295 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..b358832 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,344 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..75c76f5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,259 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..b31c739 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,304 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..6196dff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,299 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..65c69a1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,348 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..deb6ae2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,285 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..3ccb80d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,334 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..f23f2b3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,330 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..963fd2f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,383 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..7e842f1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,390 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..257f09f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,445 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..fefba8a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,281 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..36f453c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,330 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..0cdb688 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,289 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..b9ceaa5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,338 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..17ded9c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,297 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..94c1b16 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,346 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..1e0b5f0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,338 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..7e1df31 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,391 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..78d41ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,298 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..5207ae9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,351 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..863b971 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,298 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..396c5d9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,351 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..3be1e4a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,304 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..ce31457 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,357 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..9f88ec2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,302 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..e759ee7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,355 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..672c6d5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,296 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..e33bff1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,349 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..ba2f015 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..7fc62c3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,399 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..238cb6e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,302 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..348b8b0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,355 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..09e5785 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,314 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..08bb57d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,369 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..67047e6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,288 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..c4d79a7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,341 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..52da710 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,304 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..28d4704 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,357 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..0cf24da --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..302917b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,381 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..f047edf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..8e52fc6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,394 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..3583e60 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..475e3e0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,381 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..ad7cafc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..2de2e31 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,381 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..63a6a58 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..cf79327 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,381 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..46bf39b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..c2b9b4d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,381 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..750d6bb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..493a888 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,381 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..d2f8d32 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,389 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..11ee6e7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,442 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..194372a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..e9a5323 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,381 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..2545ba8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..7e55ef2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,381 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/summary.md new file mode 100644 index 0000000..632bd37 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100346/summary.md @@ -0,0 +1,151 @@ +# pipeline_setup_v3 summary + +Passed: 9/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- **v2-find-files-01-health**: llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-07-errors-catalog**: llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-09-telegram-bot-api**: llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md, docs/README.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json new file mode 100644 index 0000000..1c20004 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-01-health.json @@ -0,0 +1,295 @@ +{ + "meta": { + "case_id": "v2-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "answer_mode: expected grounded_summary, got insufficient_evidence", + "llm: expected answer to contain '/health'", + "llm: expected answer to contain 'status'", + "llm: expected answer to contain 'components'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md new file mode 100644 index 0000000..b358832 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-01-health.md @@ -0,0 +1,344 @@ +# v2-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%", + "%/health%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко объясни по документации, как работает `/health`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 96 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- answer_mode: expected grounded_summary, got insufficient_evidence +- llm: expected answer to contain '/health' +- llm: expected answer to contain 'status' +- llm: expected answer to contain 'components' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json new file mode 100644 index 0000000..75c76f5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-02-send.json @@ -0,0 +1,259 @@ +{ + "meta": { + "case_id": "v2-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'Telegram'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md new file mode 100644 index 0000000..b31c739 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-02-send.md @@ -0,0 +1,304 @@ +# v2-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%", + "%/send%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает endpoint `/send`?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": null, + "document_count": 0 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [], + "ranking_score_breakdown": [] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 59 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'Telegram' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json new file mode 100644 index 0000000..6196dff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.json @@ -0,0 +1,299 @@ +{ + "meta": { + "case_id": "v2-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/actions/{action}'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md new file mode 100644 index 0000000..65c69a1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-03-actions.md @@ -0,0 +1,348 @@ +# v2-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%", + "%/actions/{action}%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Объясни назначение `/actions/{action}`", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 390, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 40, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/actions/{action}' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..deb6ae2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.json @@ -0,0 +1,285 @@ +{ + "meta": { + "case_id": "v2-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'runtime health'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..3ccb80d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-04-runtime-health.md @@ -0,0 +1,334 @@ +# v2-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что такое runtime health в этой документации?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'runtime health' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..f23f2b3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.json @@ -0,0 +1,330 @@ +{ + "meta": { + "case_id": "v2-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'", + "llm: expected answer to contain 'interval'", + "llm: expected answer to contain 'degraded'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..963fd2f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-05-notification-loop.md @@ -0,0 +1,383 @@ +# v2-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md, docs/README.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Кратко опиши цикл отправки уведомлений", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 880, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 147 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' +- llm: expected answer to contain 'interval' +- llm: expected answer to contain 'degraded' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json new file mode 100644 index 0000000..7e842f1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.json @@ -0,0 +1,390 @@ +{ + "meta": { + "case_id": "v2-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md new file mode 100644 index 0000000..257f09f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-06-architecture.md @@ -0,0 +1,445 @@ +# v2-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как устроена архитектура Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 600, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2900, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 600, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 240, + "filename_match": 800, + "alias_match": 500, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json new file mode 100644 index 0000000..fefba8a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.json @@ -0,0 +1,281 @@ +{ + "meta": { + "case_id": "v2-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain 'worker'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md new file mode 100644 index 0000000..36f453c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-07-worker.md @@ -0,0 +1,330 @@ +# v2-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_logic_flow", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/", + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что делает worker в этом приложении?", + "profile": "docs_summary_logic_flow", + "row_count": 4, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 20, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain 'worker' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..0cdb688 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.json @@ -0,0 +1,289 @@ +{ + "meta": { + "case_id": "v2-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']", + "llm: expected answer to contain 'ok'", + "llm: expected answer to contain 'degraded'", + "llm: expected answer to contain 'unhealthy'", + "llm: expected answer to contain 'unknown'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..b9ceaa5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-08-health-statuses.md @@ -0,0 +1,338 @@ +# v2-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_domain_entity", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/", + "docs/api/", + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие статусы здоровья есть у runtime?", + "profile": "docs_summary_domain_entity", + "row_count": 4, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 410, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md'] +- llm: expected answer to contain 'ok' +- llm: expected answer to contain 'degraded' +- llm: expected answer to contain 'unhealthy' +- llm: expected answer to contain 'unknown' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json new file mode 100644 index 0000000..17ded9c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.json @@ -0,0 +1,297 @@ +{ + "meta": { + "case_id": "v2-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']", + "llm: expected answer to contain '/send'", + "llm: expected answer to contain 'message'", + "llm: expected answer to contain 'TelegramSendService'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md new file mode 100644 index 0000000..94c1b16 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-09-manual-send.md @@ -0,0 +1,346 @@ +# v2-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 4, + "llm_answer": "В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram-notify-app-overview.md.", + "answer_mode": "insufficient_evidence", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_api_endpoint", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/", + "docs/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "path_prefixes": [ + "docs/api/", + "docs/architecture/", + "docs/" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "profile": "docs_summary_api_endpoint", + "row_count": 4, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 1 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 370, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": -150 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "target_doc_missing", + "answer_mode": "insufficient_evidence" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "insufficient_evidence", + "answer_length": 131 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- llm: expected answer to contain '/send' +- llm: expected answer to contain 'message' +- llm: expected answer to contain 'TelegramSendService' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..1e0b5f0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.json @@ -0,0 +1,338 @@ +{ + "meta": { + "case_id": "v2-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got None", + "doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..7e1df31 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-docs-summary-10-telegram-integration.md @@ -0,0 +1,391 @@ +# v2-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": null, + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": null, + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 120, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got None +- doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-01-health.json new file mode 100644 index 0000000..78d41ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-01-health.json @@ -0,0 +1,298 @@ +{ + "meta": { + "case_id": "v2-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/health-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-01-health.md new file mode 100644 index 0000000..5207ae9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-01-health.md @@ -0,0 +1,351 @@ +# v2-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%health-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле описан `/health`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/health-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-02-send.json new file mode 100644 index 0000000..863b971 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-02-send.json @@ -0,0 +1,298 @@ +{ + "meta": { + "case_id": "v2-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-02-send.md new file mode 100644 index 0000000..396c5d9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-02-send.md @@ -0,0 +1,351 @@ +# v2-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramSendService" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%send-message-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл с описанием endpoint `/send`", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-03-actions.json new file mode 100644 index 0000000..3be1e4a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-03-actions.json @@ -0,0 +1,304 @@ +{ + "meta": { + "case_id": "v2-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/control-actions-endpoint.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-03-actions.md new file mode 100644 index 0000000..ce31457 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-03-actions.md @@ -0,0 +1,357 @@ +# v2-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/api/" + ], + "prefer_like_patterns": [ + "%control-actions-endpoint.md%" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где находится документация по `/actions/{action}`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 640, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/control-actions-endpoint.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json new file mode 100644 index 0000000..9f88ec2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.json @@ -0,0 +1,302 @@ +{ + "meta": { + "case_id": "v2-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md new file mode 100644 index 0000000..e759ee7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-04-runtime-health.md @@ -0,0 +1,355 @@ +# v2-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/domains/" + ], + "prefer_like_patterns": [ + "%runtime-health-entity.md%" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком документе описан runtime health?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 130, + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 650, + "match_reason": "alias_match" + }, + { + "doc": "docs/README.md", + "score": 130, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-05-loop.json new file mode 100644 index 0000000..672c6d5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-05-loop.json @@ -0,0 +1,296 @@ +{ + "meta": { + "case_id": "v2-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-05-loop.md new file mode 100644 index 0000000..e33bff1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-05-loop.md @@ -0,0 +1,349 @@ +# v2-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json new file mode 100644 index 0000000..ba2f015 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-06-architecture.json @@ -0,0 +1,344 @@ +{ + "meta": { + "case_id": "v2-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md new file mode 100644 index 0000000..7fc62c3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-06-architecture.md @@ -0,0 +1,399 @@ +# v2-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Где описана архитектура приложения?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 670, + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "alias_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 3970, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 670, + "match_reason": "alias_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 720, + "filename_match": 1600, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 50, + "path_match": 0, + "filename_match": 0, + "alias_match": 500, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json new file mode 100644 index 0000000..238cb6e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.json @@ -0,0 +1,302 @@ +{ + "meta": { + "case_id": "v2-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/errors/catalog.yaml'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md new file mode 100644 index 0000000..348b8b0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-07-errors-catalog.md @@ -0,0 +1,355 @@ +# v2-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/errors/" + ], + "prefer_like_patterns": [ + "%catalog.yaml%" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каком файле лежит каталог ошибок?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "RuntimeManager", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 150, + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 150, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 120, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 30, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/errors/catalog.yaml' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json new file mode 100644 index 0000000..09e5785 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.json @@ -0,0 +1,314 @@ +{ + "meta": { + "case_id": "v2-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md new file mode 100644 index 0000000..08bb57d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-08-docs-index.md @@ -0,0 +1,369 @@ +# v2-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/README.md\ndocs/architecture/telegram-notify-app-overview.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Покажи файл-индекс документации проекта", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/README.md" + ], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/README.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 2140, + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 300, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 10, + "path_match": 360, + "filename_match": 400, + "alias_match": 250, + "anchor_boost": 120, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 0, + "path_match": 180, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..67047e6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.json @@ -0,0 +1,288 @@ +{ + "meta": { + "case_id": "v2-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/api/send-message-endpoint.md'", + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..c4d79a7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-09-telegram-bot-api.md @@ -0,0 +1,341 @@ +# v2-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyModule" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/" + ], + "prefer_like_patterns": [], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramControlChannel", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 720, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 20, + "path_match": 180, + "filename_match": 400, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": false, + "reason": "low_confidence_shortlist", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..52da710 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.json @@ -0,0 +1,304 @@ +{ + "meta": { + "case_id": "v2-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'", + "llm: expected answer to contain 'docs/domains/runtime-health-entity.md'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..28d4704 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-find-files-10-telegram-notify-worker.md @@ -0,0 +1,357 @@ +# v2-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 3, + "llm_answer": "docs/architecture/telegram-notify-app-overview.md\ndocs/README.md", + "answer_mode": "deterministic", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "TelegramNotifyWorker" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "file_lookup", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/logic/" + ], + "prefer_like_patterns": [ + "%telegram-notification-loop.md%" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "В каких файлах описан worker `telegram_notify`?", + "profile": "file_lookup", + "row_count": 3, + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "find_files", + "primary_file": "docs/architecture/telegram-notify-app-overview.md", + "file_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 1310, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 30, + "path_match": 360, + "filename_match": 800, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 20, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 120, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "primary_file_confident", + "answer_mode": "deterministic" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "deterministic", + "answer_length": 64 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- llm: expected answer to contain 'docs/domains/runtime-health-entity.md' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json new file mode 100644 index 0000000..0cf24da --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-01-what-is-service.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md new file mode 100644 index 0000000..302917b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-01-what-is-service.md @@ -0,0 +1,381 @@ +# v2-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что это за сервис?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-02-purpose.json new file mode 100644 index 0000000..f047edf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-02-purpose.json @@ -0,0 +1,339 @@ +{ + "meta": { + "case_id": "v2-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-02-purpose.md new file mode 100644 index 0000000..8e52fc6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-02-purpose.md @@ -0,0 +1,394 @@ +# v2-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Операторские и мониторинговые клиенты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + } + ], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Для чего нужен test_echo_app?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "exact_title" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 140, + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 380, + "match_reason": "exact_title" + }, + { + "doc": "docs/README.md", + "score": 140, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 120, + "path_match": 60, + "filename_match": 200, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 140, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-03-problem.json new file mode 100644 index 0000000..3583e60 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-03-problem.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-03-problem.md new file mode 100644 index 0000000..475e3e0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-03-problem.md @@ -0,0 +1,381 @@ +# v2-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанный код", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какую задачу решает это приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json new file mode 100644 index 0000000..ad7cafc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-04-docs-contents.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md new file mode 100644 index 0000000..2de2e31 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-04-docs-contents.md @@ -0,0 +1,381 @@ +# v2-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что входит в документацию этого проекта?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-05-main-parts.json new file mode 100644 index 0000000..63a6a58 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-05-main-parts.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'систем'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-05-main-parts.md new file mode 100644 index 0000000..cf79327 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-05-main-parts.md @@ -0,0 +1,381 @@ +# v2-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие основные части есть у системы?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'систем' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json new file mode 100644 index 0000000..46bf39b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'приложен'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md new file mode 100644 index 0000000..c2b9b4d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-06-high-level-architecture.md @@ -0,0 +1,381 @@ +# v2-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Как в целом устроено приложение?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'приложен' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-07-besides-api.json new file mode 100644 index 0000000..750d6bb --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-07-besides-api.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-07-besides-api.md new file mode 100644 index 0000000..493a888 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-07-besides-api.md @@ -0,0 +1,381 @@ +# v2-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Что тут есть кроме API?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Навигация", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Связанные документы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 100, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 100, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 100, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-08-components.json new file mode 100644 index 0000000..d2f8d32 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-08-components.json @@ -0,0 +1,389 @@ +{ + "meta": { + "case_id": "v2-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE", + "llm: expected answer to contain 'Telegram Notify App'" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-08-components.md new file mode 100644 index 0000000..11ee6e7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-08-components.md @@ -0,0 +1,442 @@ +# v2-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 16, + "llm_answer": "- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).\n\nФайлы-источники:\n- docs/architecture/telegram-notify-app-overview.md\n- docs/README.md", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [ + "RuntimeManager" + ], + "doc_scope": [ + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md", + "index.test_echo_app_docs", + "docs/README.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "docs_summary_architecture", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D3_ENTITY_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%telegram-notify-app-overview.md%" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "candidate_docs_before_ranking": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Компоненты", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Интеграционные сценарии", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Telegram Bot API", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ], + "metadata_lookup": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + } + ], + "semantic": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyModule", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramNotifyWorker", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D3_ENTITY_CATALOG", + "title": "TelegramSendService", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Из чего состоит Telegram Notify App?", + "profile": "docs_summary_architecture", + "row_count": 16, + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "top_results": [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Summary", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Описание", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Границы системы", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/architecture/telegram-notify-app-overview.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + }, + "match_reason": "exact_path" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 60, + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 2140, + "match_reason": "exact_path" + }, + { + "doc": "docs/README.md", + "score": 60, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 60, + "path_match": 180, + "filename_match": 600, + "alias_match": 0, + "anchor_boost": 300, + "target_doc_boost": 1000, + "generic_penalty": 0 + } + }, + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 60, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "target_doc_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 753 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- llm: expected answer to contain 'Telegram Notify App' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-09-reading-order.json new file mode 100644 index 0000000..194372a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-09-reading-order.json @@ -0,0 +1,328 @@ +{ + "meta": { + "case_id": "v2-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": false, + "mismatches": [ + "llm: expected answer to contain 'README'" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-09-reading-order.md new file mode 100644 index 0000000..e9a5323 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-09-reading-order.md @@ -0,0 +1,381 @@ +# v2-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: False + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Правила ведения документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Ограничения", + "document_id": "architecture.telegram_notify_app" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Какие документы стоит читать сначала?", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- llm: expected answer to contain 'README' \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-10-short-summary.json new file mode 100644 index 0000000..2545ba8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-10-short-summary.json @@ -0,0 +1,326 @@ +{ + "meta": { + "case_id": "v2-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "full_chain", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + }, + { + "step": "retrieval_profile_selected", + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } + }, + { + "step": "candidate_generation", + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } + }, + { + "step": "retrieval_executed", + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + }, + { + "step": "evidence_assembled", + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } + }, + { + "step": "ranking_explained", + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } + }, + { + "step": "evidence_gate_checked", + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } + }, + { + "step": "answer_generated", + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-10-short-summary.md new file mode 100644 index 0000000..7e55ef2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/full_chain_docs_v2_matrix_v2-general-10-short-summary.md @@ -0,0 +1,381 @@ +# v2-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_04_process_v2/full_chain_docs_v2_matrix.yaml +- runner: process_v2 +- mode: full_chain +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 8, + "llm_answer": "- Purpose: точка входа в техническую документацию сервиса `test_echo_app`.\n- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health-модель и каталог ошибок.\n- Canonical structure: `docs/architecture`, `docs/api`, `docs/logic`, `docs/domains`, `docs/errors`.\n- Primary parent doc: [Архитектура Telegram Notify App](./architecture/telegram-notify-app-overview.md).\n- Navigation: документы связаны через `related_docs`, `parent`/`children` и markdown-ссылки без дублирования деталей.\n- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram.\n- Entry point: `src/telegram_notify_app/main.py`.\n- Main components: `RuntimeManager`, `TelegramControlChannel`, `TelegramNotifyModule`, `TelegramNotifyWorker`, `TelegramSendService`.\n- Configuration: `config/config.yaml` или путь из `CONFIG_PATH`.\n- Related API: [`/health`](../api/health-endpoint.md), [`/actions/{action}`](../api/control-actions-endpoint.md), [`/send`](../api/send-message-endpoint.md).\n- Related logic: [цикл отправки уведомлений](../logic/telegram-notification-loop.md).\n- Related domain: [runtime health](../domains/runtime-health-entity.md).", + "answer_mode": "grounded_summary", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [ + "index.test_echo_app_docs", + "docs/README.md", + "architecture.telegram_notify_app", + "docs/architecture/telegram-notify-app-overview.md" + ], + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +### retrieval_profile_selected +```json +{ + "input": {}, + "output": { + "profile": "general_qa_grounded_summary", + "layers": [ + "D1_DOCUMENT_CATALOG", + "D0_DOC_CHUNKS" + ], + "filters": { + "prefer_path_prefixes": [ + "docs/architecture/", + "docs/" + ], + "prefer_like_patterns": [ + "%README.md%", + "%overview%" + ], + "target_doc_hints": [] + } + } +} +``` + +### candidate_generation +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [], + "candidate_docs_before_ranking": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Описание", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Summary", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Иерархия документации", + "document_id": "index.test_echo_app_docs" + } + ], + "sources": { + "seeded": [], + "metadata_lookup": [], + "semantic": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } + } +} +``` + +### retrieval_executed +```json +{ + "input": {}, + "output": { + "query": "Дай короткое summary по проектной документации", + "profile": "general_qa_grounded_summary", + "row_count": 8, + "target_doc_hints": [], + "top_results": [ + { + "path": "docs/README.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Индекс технической документации test_echo_app", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D1_DOCUMENT_CATALOG", + "title": "Архитектура Telegram Notify App", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:История изменений", + "document_id": "architecture.telegram_notify_app" + }, + { + "path": "docs/README.md", + "layer": "D0_DOC_CHUNKS", + "title": "index.test_echo_app_docs:Обязательные правила структуры", + "document_id": "index.test_echo_app_docs" + }, + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "layer": "D0_DOC_CHUNKS", + "title": "architecture.telegram_notify_app:Контекст", + "document_id": "architecture.telegram_notify_app" + } + ] + } +} +``` + +### evidence_assembled +```json +{ + "input": {}, + "output": { + "mode": "summary", + "primary_doc": "docs/README.md", + "document_count": 2 + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/README.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + }, + "match_reason": "semantic_match" + } +} +``` + +### ranking_explained +```json +{ + "input": {}, + "output": { + "top_docs_after_ranking": [ + { + "doc": "docs/README.md", + "score": 80, + "match_reason": "semantic_match" + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score": 80, + "match_reason": "semantic_match" + } + ], + "ranking_score_breakdown": [ + { + "doc": "docs/README.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + }, + { + "doc": "docs/architecture/telegram-notify-app-overview.md", + "score_breakdown": { + "semantic": 80, + "path_match": 0, + "filename_match": 0, + "alias_match": 0, + "anchor_boost": 0, + "target_doc_boost": 0, + "generic_penalty": 0 + } + } + ] + } +} +``` + +### evidence_gate_checked +```json +{ + "input": {}, + "output": { + "passed": true, + "reason": "general_docs_found", + "answer_mode": "grounded_summary" + } +} +``` + +### answer_generated +```json +{ + "input": {}, + "output": { + "answer_mode": "grounded_summary", + "answer_length": 1170 + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/summary.md new file mode 100644 index 0000000..632bd37 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_docs_matrix/20260407_100457/summary.md @@ -0,0 +1,151 @@ +# pipeline_setup_v3 summary + +Passed: 9/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| full_chain_docs_v2_matrix.yaml | v2-general-01-what-is-service | full_chain | Что это за сервис? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-02-purpose | full_chain | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-03-problem | full_chain | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-04-docs-contents | full_chain | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-05-main-parts | full_chain | Какие основные части есть у системы? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-06-high-level-architecture | full_chain | Как в целом устроено приложение? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-07-besides-api | full_chain | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-general-08-components | full_chain | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-09-reading-order | full_chain | Какие документы стоит читать сначала? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-general-10-short-summary | full_chain | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-01-health | full_chain | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-02-send | full_chain | Что делает endpoint `/send`? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-03-actions | full_chain | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-04-runtime-health | full_chain | Что такое runtime health в этой документации? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-05-notification-loop | full_chain | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-06-architecture | full_chain | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-07-worker | full_chain | Что делает worker в этом приложении? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-08-health-statuses | full_chain | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-09-manual-send | full_chain | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-docs-summary-10-telegram-integration | full_chain | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-01-health | full_chain | В каком файле описан `/health`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-02-send | full_chain | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-03-actions | full_chain | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-04-runtime-health | full_chain | В каком документе описан runtime health? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-05-loop | full_chain | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-06-architecture | full_chain | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-07-errors-catalog | full_chain | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-08-docs-index | full_chain | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-09-telegram-bot-api | full_chain | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✗ | +| full_chain_docs_v2_matrix.yaml | v2-find-files-10-telegram-notify-worker | full_chain | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✗ | + +## Failures +- **v2-general-05-main-parts**: llm: expected answer to contain 'систем' +- **v2-general-06-high-level-architecture**: llm: expected answer to contain 'приложен' +- **v2-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE; llm: expected answer to contain 'Telegram Notify App' +- **v2-general-09-reading-order**: llm: expected answer to contain 'README' +- **v2-docs-summary-01-health**: answer_mode: expected grounded_summary, got insufficient_evidence; llm: expected answer to contain '/health'; llm: expected answer to contain 'status'; llm: expected answer to contain 'components' +- **v2-docs-summary-02-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'Telegram' +- **v2-docs-summary-03-actions**: doc_scope should contain 'docs/api/control-actions-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/actions/{action}' +- **v2-docs-summary-04-runtime-health**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'runtime health' +- **v2-docs-summary-05-notification-loop**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker'; llm: expected answer to contain 'interval'; llm: expected answer to contain 'degraded' +- **v2-docs-summary-07-worker**: doc_scope should contain 'docs/logic/telegram-notification-loop.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain 'worker' +- **v2-docs-summary-08-health-statuses**: doc_scope should contain 'docs/domains/runtime-health-entity.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; doc_scope should contain 'docs/api/health-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md']; llm: expected answer to contain 'ok'; llm: expected answer to contain 'degraded'; llm: expected answer to contain 'unhealthy'; llm: expected answer to contain 'unknown' +- **v2-docs-summary-09-manual-send**: doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md']; llm: expected answer to contain '/send'; llm: expected answer to contain 'message'; llm: expected answer to contain 'TelegramSendService' +- **v2-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got None; doc_scope should contain 'docs/api/send-message-endpoint.md', got ['architecture.telegram_notify_app', 'docs/architecture/telegram-notify-app-overview.md', 'index.test_echo_app_docs', 'docs/README.md'] +- **v2-find-files-01-health**: llm: expected answer to contain 'docs/api/health-endpoint.md' +- **v2-find-files-02-send**: llm: expected answer to contain 'docs/api/send-message-endpoint.md' +- **v2-find-files-03-actions**: llm: expected answer to contain 'docs/api/control-actions-endpoint.md' +- **v2-find-files-04-runtime-health**: llm: expected answer to contain 'docs/domains/runtime-health-entity.md' +- **v2-find-files-05-loop**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-07-errors-catalog**: llm: expected answer to contain 'docs/errors/catalog.yaml' +- **v2-find-files-09-telegram-bot-api**: llm: expected answer to contain 'docs/api/send-message-endpoint.md'; llm: expected answer to contain 'docs/logic/telegram-notification-loop.md' +- **v2-find-files-10-telegram-notify-worker**: llm: expected answer to contain 'docs/logic/telegram-notification-loop.md'; llm: expected answer to contain 'docs/domains/runtime-health-entity.md' + +## LLM Answers +- **v2-general-01-what-is-service** + Query: Что это за сервис? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-02-purpose** + Query: Для чего нужен test_echo_app? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-general-03-problem** + Query: Какую задачу решает это приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-04-docs-contents** + Query: Что входит в документацию этого проекта? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-05-main-parts** + Query: Какие основные части есть у системы? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-06-high-level-architecture** + Query: Как в целом устроено приложение? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-07-besides-api** + Query: Что тут есть кроме API? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-08-components** + Query: Из чего состоит Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-general-09-reading-order** + Query: Какие документы стоит читать сначала? + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-general-10-short-summary** + Query: Дай короткое summary по проектной документации + > \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/architecture\`, \`docs/api\`, \`docs/logic\`, \`docs/domains\`, \`docs/errors\`. \- Primary parent doc: \[Архитектура Telegram Notify App\]\(./architecture/telegram\-notify\-app\-overview.md\). \- Navigation: документы связаны через \`related\_docs\`, \`parent\`/\`children\` и markdown\-ссылки без дублирования деталей. \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`… +- **v2-docs-summary-01-health** + Query: Кратко объясни по документации, как работает `/health` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/README.md. +- **v2-docs-summary-02-send** + Query: Что делает endpoint `/send`? + > В поднятом контексте не найден целевой документ по запросу. +- **v2-docs-summary-03-actions** + Query: Объясни назначение `/actions/{action}` + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-04-runtime-health** + Query: Что такое runtime health в этой документации? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-05-notification-loop** + Query: Кратко опиши цикл отправки уведомлений + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md, docs/README.md. +- **v2-docs-summary-06-architecture** + Query: Как устроена архитектура Telegram Notify App? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). Файлы\-источники: \- docs/architecture/telegram\-notify\-app\-overview.md \- docs/README.md +- **v2-docs-summary-07-worker** + Query: Что делает worker в этом приложении? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-08-health-statuses** + Query: Какие статусы здоровья есть у runtime? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-09-manual-send** + Query: Как в приложении устроена ручная отправка сообщения? + > В поднятом контексте не найден целевой документ по запросу. Ближайшие документы: docs/architecture/telegram\-notify\-app\-overview.md. +- **v2-docs-summary-10-telegram-integration** + Query: Как сервис взаимодействует с Telegram в целом? + > \- Purpose: сервис поднимает HTTP control plane и фоновый worker для отправки уведомлений в Telegram. \- Entry point: \`src/telegram\_notify\_app/main.py\`. \- Main components: \`RuntimeManager\`, \`TelegramControlChannel\`, \`TelegramNotifyModule\`, \`TelegramNotifyWorker\`, \`TelegramSendService\`. \- Configuration: \`config/config.yaml\` или путь из \`CONFIG\_PATH\`. \- Related API: \[\`/health\`\]\(../api/health\-endpoint.md\), \[\`/actions/\{action\}\`\]\(../api/control\-actions\-endpoint.md\), \[\`/send\`\]\(../api/send\-message\-endpoint.md\). \- Related logic: \[цикл отправки уведомлений\]\(../logic/telegram\-notification\-loop.md\). \- Related domain: \[runtime health\]\(../domains/runtime\-health\-entity.md\). \- Purpose: точка входа в техническую документацию сервиса \`test\_echo\_app\`. \- Scope: архитектура, HTTP API control plane, цикл отправки уведомлений, health\-модель и каталог ошибок. \- Canonical structure: \`docs/arc… +- **v2-find-files-01-health** + Query: В каком файле описан `/health`? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-02-send** + Query: Покажи файл с описанием endpoint `/send` + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-03-actions** + Query: Где находится документация по `/actions/{action}`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-04-runtime-health** + Query: В каком документе описан runtime health? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-05-loop** + Query: Найди файл с логикой цикла отправки уведомлений + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-06-architecture** + Query: Где описана архитектура приложения? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-07-errors-catalog** + Query: В каком файле лежит каталог ошибок? + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-08-docs-index** + Query: Покажи файл-индекс документации проекта + > docs/README.md docs/architecture/telegram\-notify\-app\-overview.md +- **v2-find-files-09-telegram-bot-api** + Query: Какие файлы относятся к Telegram Bot API интеграции? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md +- **v2-find-files-10-telegram-notify-worker** + Query: В каких файлах описан worker `telegram_notify`? + > docs/architecture/telegram\-notify\-app\-overview.md docs/README.md \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..c7a8098 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..cf898c9 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..cc0f481 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..c3fafc4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..4870cc7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,63 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..a5b5d46 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,82 @@ +# v2-router-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..35bc7a6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..468082f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..7141f0d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..ef11dd8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..983cfdd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..221648a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..0895b26 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,57 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..701a79d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,76 @@ +# v2-router-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "LOGIC_FLOW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "LOGIC_FLOW", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..5c8de85 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..9356b42 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "DOMAIN_ENTITY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "DOMAIN_ENTITY", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..f8d1581 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..e3493ca --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "API_ENDPOINT", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "API_ENDPOINT", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..3dbee41 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,57 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA", + "anchor_type: expected API_ENDPOINT, got GENERAL_OVERVIEW" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 0.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..6f4f175 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,74 @@ +# v2-router-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 0.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA +- anchor_type: expected API_ENDPOINT, got GENERAL_OVERVIEW \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..66a48dc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..de59127 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,78 @@ +# v2-router-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..507ac1d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..b88507f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,78 @@ +# v2-router-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..f38a120 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,63 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..bf3de40 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,82 @@ +# v2-router-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..d7bd833 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..5b99592 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,80 @@ +# v2-router-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..6e37615 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,57 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..af50f94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,76 @@ +# v2-router-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..5726d29 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..d828f82 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,80 @@ +# v2-router-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..d5cfe93 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..187c811 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,80 @@ +# v2-router-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..597c4b1 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..3c765c0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,80 @@ +# v2-router-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..55b00b3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..ab45131 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,72 @@ +# v2-router-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "FIND_FILES", + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..740d9ee --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,57 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..2035d8c --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,76 @@ +# v2-router-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "anchor_type": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..0731458 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..34cfd50 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,72 @@ +# v2-router-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..b8d2ba7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..d11b972 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,72 @@ +# v2-router-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..3038ebf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..4b1bfc5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,72 @@ +# v2-router-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..eae1f2f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..f4877f3 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,72 @@ +# v2-router-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..1f79992 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..bf5a89d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,72 @@ +# v2-router-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..44292ab --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..c5ae515 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,72 @@ +# v2-router-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..4b21fbc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..7e312de --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,72 @@ +# v2-router-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..aec7aba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN", + "anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..a77397e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,78 @@ +# v2-router-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "anchor_type": "ARCHITECTURE", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": "ARCHITECTURE", + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN +- anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..da4f134 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..a41a25b --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,72 @@ +# v2-router-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..38933e5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..e5a6141 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,72 @@ +# v2-router-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "anchor_type": "GENERAL_OVERVIEW", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "anchor_type": null, + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/summary.md new file mode 100644 index 0000000..a562817 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router/20260407_102851/summary.md @@ -0,0 +1,40 @@ +# pipeline_setup_v3 summary + +Passed: 28/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| router_only_docs_v2_matrix.yaml | v2-router-general-01-what-is-service | router_only | Что это за сервис? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-02-purpose | router_only | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-03-problem | router_only | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-04-docs-contents | router_only | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-05-main-parts | router_only | Какие основные части есть у системы? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-06-high-level-architecture | router_only | Как в целом устроено приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-07-besides-api | router_only | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-08-components | router_only | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-general-09-reading-order | router_only | Какие документы стоит читать сначала? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-10-short-summary | router_only | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-01-health | router_only | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-02-send | router_only | Что делает endpoint `/send`? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-03-actions | router_only | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-04-runtime-health | router_only | Что такое runtime health в этой документации? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-05-notification-loop | router_only | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-06-architecture | router_only | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-07-worker | router_only | Что делает worker в этом приложении? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-08-health-statuses | router_only | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-09-manual-send | router_only | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-10-telegram-integration | router_only | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-01-health | router_only | В каком файле описан `/health`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-02-send | router_only | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-03-actions | router_only | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-04-runtime-health | router_only | В каком документе описан runtime health? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-05-loop | router_only | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-06-architecture | router_only | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-07-errors-catalog | router_only | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-08-docs-index | router_only | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-09-telegram-bot-api | router_only | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-10-telegram-notify-worker | router_only | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✓ | + +## Failures +- **v2-router-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN; anchor_type: expected GENERAL_OVERVIEW, got ARCHITECTURE +- **v2-router-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA; anchor_type: expected API_ENDPOINT, got GENERAL_OVERVIEW \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..9ad57ff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..586d514 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..5c865f7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..6d4f87a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..f2c9a53 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,63 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..5465b2e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,82 @@ +# v2-router-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..4427154 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..a26e3e2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..06d5e1a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..5f7abaf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..87f19ff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..2c84c78 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..953a50f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,57 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..006c102 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,76 @@ +# v2-router-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..3286890 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..7cf2f0e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..0545f4e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..10c0e8a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..9d0de5d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,54 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 0.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..f24ae89 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,71 @@ +# v2-router-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 0.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..61ddbb0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..e208c21 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,79 @@ +# v2-router-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..dcada94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..255d636 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,79 @@ +# v2-router-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..5033f00 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,64 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..5231b00 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,83 @@ +# v2-router-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..6cc99f4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,62 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..6857b04 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,81 @@ +# v2-router-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..77a8f94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,58 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..d73b60d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,77 @@ +# v2-router-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..9927fce --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,62 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..960d392 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,81 @@ +# v2-router-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..1932366 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..90c1386 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,80 @@ +# v2-router-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..6b8c3ee --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..bc2fe36 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,80 @@ +# v2-router-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..fb33239 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..c5e6a01 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,72 @@ +# v2-router-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..d535a15 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,58 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..c7699bc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,77 @@ +# v2-router-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..d81ef16 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..1eb13a5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,70 @@ +# v2-router-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..a30bce2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..1836903 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,70 @@ +# v2-router-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..1918311 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..6a95066 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,70 @@ +# v2-router-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..9b56912 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..38ec7f8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,70 @@ +# v2-router-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..a009a7f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..e4c3ddf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,70 @@ +# v2-router-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..18664f6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..915cfdd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,70 @@ +# v2-router-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..42d1338 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..9edb373 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,70 @@ +# v2-router-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..8310d99 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..9cd83a6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,77 @@ +# v2-router-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..29f98ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..b1f1028 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,70 @@ +# v2-router-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..3405b11 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..1a8dd33 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,70 @@ +# v2-router-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/summary.md new file mode 100644 index 0000000..8e30d1e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_modules/20260407_114412/summary.md @@ -0,0 +1,40 @@ +# pipeline_setup_v3 summary + +Passed: 28/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| router_only_docs_v2_matrix.yaml | v2-router-general-01-what-is-service | router_only | Что это за сервис? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-02-purpose | router_only | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-03-problem | router_only | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-04-docs-contents | router_only | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-05-main-parts | router_only | Какие основные части есть у системы? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-06-high-level-architecture | router_only | Как в целом устроено приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-07-besides-api | router_only | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-08-components | router_only | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-general-09-reading-order | router_only | Какие документы стоит читать сначала? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-10-short-summary | router_only | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-01-health | router_only | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-02-send | router_only | Что делает endpoint `/send`? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-03-actions | router_only | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-04-runtime-health | router_only | Что такое runtime health в этой документации? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-05-notification-loop | router_only | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-06-architecture | router_only | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-07-worker | router_only | Что делает worker в этом приложении? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-08-health-statuses | router_only | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-09-manual-send | router_only | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-10-telegram-integration | router_only | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-01-health | router_only | В каком файле описан `/health`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-02-send | router_only | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-03-actions | router_only | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-04-runtime-health | router_only | В каком документе описан runtime health? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-05-loop | router_only | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-06-architecture | router_only | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-07-errors-catalog | router_only | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-08-docs-index | router_only | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-09-telegram-bot-api | router_only | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-10-telegram-notify-worker | router_only | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✓ | + +## Failures +- **v2-router-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN +- **v2-router-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..9ad57ff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..586d514 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..5c865f7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..6d4f87a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..f2c9a53 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,63 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..5465b2e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,82 @@ +# v2-router-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..4427154 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..a26e3e2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..06d5e1a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..5f7abaf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..87f19ff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..2c84c78 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..953a50f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,57 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..006c102 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,76 @@ +# v2-router-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..3286890 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..7cf2f0e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..0545f4e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..10c0e8a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..9d0de5d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,54 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 0.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..f24ae89 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,71 @@ +# v2-router-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 0.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..61ddbb0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..e208c21 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,79 @@ +# v2-router-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..dcada94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..255d636 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,79 @@ +# v2-router-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..5033f00 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,64 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..5231b00 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,83 @@ +# v2-router-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..6cc99f4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,62 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..6857b04 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,81 @@ +# v2-router-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..77a8f94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,58 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..d73b60d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,77 @@ +# v2-router-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..9927fce --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,62 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..960d392 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,81 @@ +# v2-router-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..1932366 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..90c1386 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,80 @@ +# v2-router-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..6b8c3ee --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..bc2fe36 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,80 @@ +# v2-router-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..fb33239 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..c5e6a01 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,72 @@ +# v2-router-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..d535a15 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,58 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..c7699bc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,77 @@ +# v2-router-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..d81ef16 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..1eb13a5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,70 @@ +# v2-router-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..a30bce2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..1836903 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,70 @@ +# v2-router-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..1918311 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..6a95066 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,70 @@ +# v2-router-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..9b56912 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..38ec7f8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,70 @@ +# v2-router-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..a009a7f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..e4c3ddf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,70 @@ +# v2-router-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..18664f6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..915cfdd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,70 @@ +# v2-router-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..42d1338 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..9edb373 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,70 @@ +# v2-router-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..8310d99 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..9cd83a6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,77 @@ +# v2-router-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..29f98ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..b1f1028 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,70 @@ +# v2-router-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..3405b11 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..1a8dd33 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,70 @@ +# v2-router-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/summary.md new file mode 100644 index 0000000..8e30d1e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_no_anchor_type/20260407_111829/summary.md @@ -0,0 +1,40 @@ +# pipeline_setup_v3 summary + +Passed: 28/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| router_only_docs_v2_matrix.yaml | v2-router-general-01-what-is-service | router_only | Что это за сервис? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-02-purpose | router_only | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-03-problem | router_only | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-04-docs-contents | router_only | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-05-main-parts | router_only | Какие основные части есть у системы? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-06-high-level-architecture | router_only | Как в целом устроено приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-07-besides-api | router_only | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-08-components | router_only | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-general-09-reading-order | router_only | Какие документы стоит читать сначала? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-10-short-summary | router_only | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-01-health | router_only | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-02-send | router_only | Что делает endpoint `/send`? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-03-actions | router_only | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-04-runtime-health | router_only | Что такое runtime health в этой документации? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-05-notification-loop | router_only | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-06-architecture | router_only | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-07-worker | router_only | Что делает worker в этом приложении? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-08-health-statuses | router_only | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-09-manual-send | router_only | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-10-telegram-integration | router_only | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-01-health | router_only | В каком файле описан `/health`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-02-send | router_only | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-03-actions | router_only | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-04-runtime-health | router_only | В каком документе описан runtime health? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-05-loop | router_only | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-06-architecture | router_only | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-07-errors-catalog | router_only | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-08-docs-index | router_only | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-09-telegram-bot-api | router_only | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-10-telegram-notify-worker | router_only | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✓ | + +## Failures +- **v2-router-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN +- **v2-router-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..9ad57ff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..586d514 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..5c865f7 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,59 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..6d4f87a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,78 @@ +# v2-router-docs-summary-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..f2c9a53 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,63 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..5465b2e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,82 @@ +# v2-router-docs-summary-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..4427154 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..a26e3e2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..06d5e1a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..5f7abaf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-05-notification-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notification-loop" + ], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..87f19ff --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..2c84c78 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..953a50f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,57 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..006c102 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,76 @@ +# v2-router-docs-summary-07-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..3286890 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..7cf2f0e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-08-health-statuses + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..0545f4e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..10c0e8a --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,80 @@ +# v2-router-docs-summary-09-manual-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..9d0de5d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,54 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA" + ], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 0.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..f24ae89 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,71 @@ +# v2-router-docs-summary-10-telegram-integration + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 0.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..61ddbb0 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..e208c21 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,79 @@ +# v2-router-find-files-01-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..dcada94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..255d636 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,79 @@ +# v2-router-find-files-02-send + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..5033f00 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,64 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..5231b00 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,83 @@ +# v2-router-find-files-03-actions + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "API_ENDPOINT", + "FIND_FILES" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..6cc99f4 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,62 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "DOMAIN_ENTITY", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..6857b04 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,81 @@ +# v2-router-find-files-04-runtime-health + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "DOMAIN_ENTITY", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "runtime-health-entity" + ], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..77a8f94 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,58 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..d73b60d --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,77 @@ +# v2-router-find-files-05-loop + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..9927fce --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,62 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..960d392 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,81 @@ +# v2-router-find-files-06-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE", + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "telegram-notify-app-overview" + ], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..1932366 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..90c1386 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,80 @@ +# v2-router-find-files-07-errors-catalog + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "catalog-errors" + ], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..6b8c3ee --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,61 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..bc2fe36 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,80 @@ +# v2-router-find-files-08-docs-index + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [ + "docs-readme" + ], + "target_doc_hints": [ + "docs/README.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..fb33239 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,53 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..c5e6a01 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,72 @@ +# v2-router-find-files-09-telegram-bot-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..d535a15 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,58 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..c7699bc --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,77 @@ +# v2-router-find-files-10-telegram-notify-worker + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "FIND_FILES", + "LOGIC_FLOW" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..d81ef16 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..1eb13a5 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,70 @@ +# v2-router-general-01-what-is-service + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что это за сервис? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..a30bce2 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..1836903 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,70 @@ +# v2-router-general-02-purpose + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..1918311 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..6a95066 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,70 @@ +# v2-router-general-03-problem + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..9b56912 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..38ec7f8 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,70 @@ +# v2-router-general-04-docs-contents + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..a009a7f --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..e4c3ddf --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,70 @@ +# v2-router-general-05-main-parts + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..18664f6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..915cfdd --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,70 @@ +# v2-router-general-06-high-level-architecture + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..42d1338 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..9edb373 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,70 @@ +# v2-router-general-07-besides-api + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..8310d99 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,60 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN" + ], + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..9cd83a6 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,77 @@ +# v2-router-general-08-components + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [ + "ARCHITECTURE" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..29f98ba --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..b1f1028 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,70 @@ +# v2-router-general-09-reading-order + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..3405b11 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,51 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "runner": "process_v2", + "mode": "router_only", + "passed": true, + "mismatches": [], + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} + } + }, + "pipeline_steps": [ + { + "step": "router_resolved", + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } + }, + { + "step": "anchors_extracted", + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } + }, + { + "step": "alias_resolution", + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } + } + ] +} \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..1a8dd33 --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,70 @@ +# v2-router-general-10-short-summary + +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v3/cases/suite_05_process_v2_intent_router/router_only_docs_v2_matrix.yaml +- runner: process_v2 +- mode: router_only +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "rag_count": 0, + "llm_answer": "", + "answer_mode": "partial", + "path_scope": [], + "symbol_candidates": [], + "entity_candidates": [], + "doc_scope": [], + "layers": [], + "filters": {} +} + +## Pipeline Steps +### router_resolved +```json +{ + "input": {}, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "confidence": 1.0 + } +} +``` + +### anchors_extracted +```json +{ + "input": {}, + "output": { + "signal_types": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [] + } +} +``` + +### alias_resolution +```json +{ + "input": {}, + "output": { + "resolved_aliases": [], + "target_doc_hints": [] + } +} +``` + +## Diagnostics +{} + + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/summary.md b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/summary.md new file mode 100644 index 0000000..8e30d1e --- /dev/null +++ b/tests/pipeline_setup_v3/test_results/process_v2_intent_router_refactored/20260407_113944/summary.md @@ -0,0 +1,40 @@ +# pipeline_setup_v3 summary + +Passed: 28/30 + +| File | Case | Mode | Query | Actual sub-intent | RAG layers | Tokens | Pass | +|------|------|------|-------|-------------------|------------|--------|------| +| router_only_docs_v2_matrix.yaml | v2-router-general-01-what-is-service | router_only | Что это за сервис? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-02-purpose | router_only | Для чего нужен test_echo_app? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-03-problem | router_only | Какую задачу решает это приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-04-docs-contents | router_only | Что входит в документацию этого проекта? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-05-main-parts | router_only | Какие основные части есть у системы? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-06-high-level-architecture | router_only | Как в целом устроено приложение? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-07-besides-api | router_only | Что тут есть кроме API? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-08-components | router_only | Из чего состоит Telegram Notify App? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-general-09-reading-order | router_only | Какие документы стоит читать сначала? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-general-10-short-summary | router_only | Дай короткое summary по проектной документации | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-01-health | router_only | Кратко объясни по документации, как работает `/health` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-02-send | router_only | Что делает endpoint `/send`? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-03-actions | router_only | Объясни назначение `/actions/{action}` | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-04-runtime-health | router_only | Что такое runtime health в этой документации? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-05-notification-loop | router_only | Кратко опиши цикл отправки уведомлений | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-06-architecture | router_only | Как устроена архитектура Telegram Notify App? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-07-worker | router_only | Что делает worker в этом приложении? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-08-health-statuses | router_only | Какие статусы здоровья есть у runtime? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-09-manual-send | router_only | Как в приложении устроена ручная отправка сообщения? | SUMMARY | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-docs-summary-10-telegram-integration | router_only | Как сервис взаимодействует с Telegram в целом? | SUMMARY | — | — | ✗ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-01-health | router_only | В каком файле описан `/health`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-02-send | router_only | Покажи файл с описанием endpoint `/send` | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-03-actions | router_only | Где находится документация по `/actions/{action}`? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-04-runtime-health | router_only | В каком документе описан runtime health? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-05-loop | router_only | Найди файл с логикой цикла отправки уведомлений | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-06-architecture | router_only | Где описана архитектура приложения? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-07-errors-catalog | router_only | В каком файле лежит каталог ошибок? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-08-docs-index | router_only | Покажи файл-индекс документации проекта | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-09-telegram-bot-api | router_only | Какие файлы относятся к Telegram Bot API интеграции? | FIND_FILES | — | — | ✓ | +| router_only_docs_v2_matrix.yaml | v2-router-find-files-10-telegram-notify-worker | router_only | В каких файлах описан worker `telegram_notify`? | FIND_FILES | — | — | ✓ | + +## Failures +- **v2-router-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN +- **v2-router-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v4/README.md b/tests/pipeline_setup_v4/README.md new file mode 100644 index 0000000..057c1c4 --- /dev/null +++ b/tests/pipeline_setup_v4/README.md @@ -0,0 +1,25 @@ +# pipeline_setup_v4 + +`pipeline_setup_v4` is a component-level test harness built on top of the `v3` ideas. + +Differences from `v3`: + +- each YAML case targets a single isolated component; +- results are written next to the suite in `cases/.../test_runs/...`; +- the first supported component is `process_v2_intent_router`. + +## Run + +```bash +PYTHONPATH=. python -m tests.pipeline_setup_v4.run \ + --cases-dir tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml \ + --run-name smoke_v4 +``` + +LLM-first router suite: + +```bash +PYTHONPATH=. python -m tests.pipeline_setup_v4.run \ + --cases-dir tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml \ + --run-name llm_first_v3 +``` diff --git a/tests/pipeline_setup_v4/__init__.py b/tests/pipeline_setup_v4/__init__.py new file mode 100644 index 0000000..d8578dc --- /dev/null +++ b/tests/pipeline_setup_v4/__init__.py @@ -0,0 +1 @@ +"""pipeline_setup_v4 package.""" diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml new file mode 100644 index 0000000..030d93c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml @@ -0,0 +1,243 @@ +defaults: + component: process_v2_intent_router + +cases: + - id: v2-router-general-01-what-is-service + query: "Что это за сервис?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-02-purpose + query: "Для чего нужен test_echo_app?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-03-problem + query: "Какую задачу решает это приложение?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-04-docs-contents + query: "Что входит в документацию этого проекта?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-05-main-parts + query: "Какие основные части есть у системы?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-06-high-level-architecture + query: "Как в целом устроено приложение?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-07-besides-api + query: "Что тут есть кроме API?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-08-components + query: "Из чего состоит Telegram Notify App?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-general-09-reading-order + query: "Какие документы стоит читать сначала?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-general-10-short-summary + query: "Дай короткое summary по проектной документации" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-docs-summary-01-health + query: "Кратко объясни по документации, как работает `/health`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-02-send + query: "Что делает endpoint `/send`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-03-actions + query: "Объясни назначение `/actions/{action}`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-04-runtime-health + query: "Что такое runtime health в этой документации?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-05-notification-loop + query: "Кратко опиши цикл отправки уведомлений" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-06-architecture + query: "Как устроена архитектура Telegram Notify App?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-07-worker + query: "Что делает worker в этом приложении?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-08-health-statuses + query: "Какие статусы здоровья есть у runtime?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-09-manual-send + query: "Как в приложении устроена ручная отправка сообщения?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v2-router-docs-summary-10-telegram-integration + query: "Как сервис взаимодействует с Telegram в целом?" + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v2-router-find-files-01-health + query: "В каком файле описан `/health`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-02-send + query: "Покажи файл с описанием endpoint `/send`" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-03-actions + query: "Где находится документация по `/actions/{action}`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-04-runtime-health + query: "В каком документе описан runtime health?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-05-loop + query: "Найди файл с логикой цикла отправки уведомлений" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-06-architecture + query: "Где описана архитектура приложения?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-07-errors-catalog + query: "В каком файле лежит каталог ошибок?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-08-docs-index + query: "Покажи файл-индекс документации проекта" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-09-telegram-bot-api + query: "Какие файлы относятся к Telegram Bot API интеграции?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v2-router-find-files-10-telegram-notify-worker + query: "В каких файлах описан worker `telegram_notify`?" + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..e0ce050 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..4d811c8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,91 @@ +# v2-router-docs-summary-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..1317d91 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..d70e7b6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,91 @@ +# v2-router-docs-summary-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..2907bc2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..8d1e4cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,95 @@ +# v2-router-docs-summary-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..28a51bc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..95d4c46 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,93 @@ +# v2-router-docs-summary-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..52760be --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,90 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA" + ] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..8673dfa --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,98 @@ +# v2-router-docs-summary-05-notification-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: False + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..30385d9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..401cc55 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,105 @@ +# v2-router-docs-summary-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..ed32c20 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..397ca47 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,89 @@ +# v2-router-docs-summary-07-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..337ae24 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,88 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA" + ] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..5a07b34 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,96 @@ +# v2-router-docs-summary-08-health-statuses + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: False + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..15af6d7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,90 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA" + ] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..ca816f5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,98 @@ +# v2-router-docs-summary-09-manual-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: False + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..78ed8cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..a1bdb67 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,89 @@ +# v2-router-docs-summary-10-telegram-integration + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..e9cf5a9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..c53bee3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,93 @@ +# v2-router-find-files-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..b100875 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..00d0a8b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,91 @@ +# v2-router-find-files-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..2e1f88e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..7d725f4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,95 @@ +# v2-router-find-files-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..6bcb3fe --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..67a4971 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,97 @@ +# v2-router-find-files-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..22e0dc9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..e5fb3cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,89 @@ +# v2-router-find-files-05-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..0274ae5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..e9e5561 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,93 @@ +# v2-router-find-files-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..f395d9e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..06d4c48 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,97 @@ +# v2-router-find-files-07-errors-catalog + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..f9b1b13 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..fa65c6a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,91 @@ +# v2-router-find-files-08-docs-index + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..a8ff1d8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": false, + "mismatches": [ + "sub_intent: expected FIND_FILES, got SUMMARY" + ] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..3450321 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,95 @@ +# v2-router-find-files-09-telegram-bot-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: False + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- sub_intent: expected FIND_FILES, got SUMMARY \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..982e56f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..5f77518 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,95 @@ +# v2-router-find-files-10-telegram-notify-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..27d18f8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..aa7263c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,79 @@ +# v2-router-general-01-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..4480103 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..d662352 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,89 @@ +# v2-router-general-02-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..d6694a2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..e0055a8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,85 @@ +# v2-router-general-03-problem + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..cf9f2ed --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..adba625 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,83 @@ +# v2-router-general-04-docs-contents + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..bb52214 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..4dd636a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,85 @@ +# v2-router-general-05-main-parts + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..b5a6df4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..693f1cf --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,83 @@ +# v2-router-general-06-high-level-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..1336596 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..72b9d53 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,89 @@ +# v2-router-general-07-besides-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..e7aa4e4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,89 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..8973686 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,99 @@ +# v2-router-general-08-components + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Из чего состоит Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..0b62adf --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..a246294 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,83 @@ +# v2-router-general-09-reading-order + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..2b70a92 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..2df755e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,81 @@ +# v2-router-general-10-short-summary + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/summary.md new file mode 100644 index 0000000..b7d961f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_124956/summary.md @@ -0,0 +1,42 @@ +# pipeline_setup_v4 summary + +Passed: 26/30 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v2-router-general-01-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-02-purpose | process_v2_intent_router | Для чего нужен test_echo_app? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-03-problem | process_v2_intent_router | Какую задачу решает это приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-04-docs-contents | process_v2_intent_router | Что входит в документацию этого проекта? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-05-main-parts | process_v2_intent_router | Какие основные части есть у системы? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-06-high-level-architecture | process_v2_intent_router | Как в целом устроено приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-07-besides-api | process_v2_intent_router | Что тут есть кроме API? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-08-components | process_v2_intent_router | Из чего состоит Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-general-09-reading-order | process_v2_intent_router | Какие документы стоит читать сначала? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-10-short-summary | process_v2_intent_router | Дай короткое summary по проектной документации | GENERAL_QA | SUMMARY | ✓ | +| v2-router-docs-summary-01-health | process_v2_intent_router | Кратко объясни по документации, как работает `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-02-send | process_v2_intent_router | Что делает endpoint `/send`? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-03-actions | process_v2_intent_router | Объясни назначение `/actions/{action}` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-04-runtime-health | process_v2_intent_router | Что такое runtime health в этой документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-05-notification-loop | process_v2_intent_router | Кратко опиши цикл отправки уведомлений | GENERAL_QA | SUMMARY | ✗ | +| v2-router-docs-summary-06-architecture | process_v2_intent_router | Как устроена архитектура Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-07-worker | process_v2_intent_router | Что делает worker в этом приложении? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-08-health-statuses | process_v2_intent_router | Какие статусы здоровья есть у runtime? | GENERAL_QA | SUMMARY | ✗ | +| v2-router-docs-summary-09-manual-send | process_v2_intent_router | Как в приложении устроена ручная отправка сообщения? | GENERAL_QA | SUMMARY | ✗ | +| v2-router-docs-summary-10-telegram-integration | process_v2_intent_router | Как сервис взаимодействует с Telegram в целом? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-find-files-01-health | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-02-send | process_v2_intent_router | Покажи файл с описанием endpoint `/send` | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-03-actions | process_v2_intent_router | Где находится документация по `/actions/{action}`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-04-runtime-health | process_v2_intent_router | В каком документе описан runtime health? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-05-loop | process_v2_intent_router | Найди файл с логикой цикла отправки уведомлений | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-06-architecture | process_v2_intent_router | Где описана архитектура приложения? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-07-errors-catalog | process_v2_intent_router | В каком файле лежит каталог ошибок? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-08-docs-index | process_v2_intent_router | Покажи файл-индекс документации проекта | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-09-telegram-bot-api | process_v2_intent_router | Какие файлы относятся к Telegram Bot API интеграции? | DOC_EXPLAIN | SUMMARY | ✗ | +| v2-router-find-files-10-telegram-notify-worker | process_v2_intent_router | В каких файлах описан worker `telegram_notify`? | DOC_EXPLAIN | FIND_FILES | ✓ | + +## Failures +- **v2-router-docs-summary-05-notification-loop**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA +- **v2-router-docs-summary-08-health-statuses**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA +- **v2-router-docs-summary-09-manual-send**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA +- **v2-router-find-files-09-telegram-bot-api**: sub_intent: expected FIND_FILES, got SUMMARY \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..e0ce050 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..4d811c8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,91 @@ +# v2-router-docs-summary-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "кратко", + "работает" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..1317d91 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..d70e7b6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,91 @@ +# v2-router-docs-summary-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..2907bc2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..8d1e4cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,95 @@ +# v2-router-docs-summary-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/actions/{action}", + "назначение", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..28a51bc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..95d4c46 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,93 @@ +# v2-router-docs-summary-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..aba1991 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..19ea1b6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,97 @@ +# v2-router-docs-summary-05-notification-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "telegram-notify-loop", + "кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..30385d9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..401cc55 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,105 @@ +# v2-router-docs-summary-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..ed32c20 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..397ca47 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,89 @@ +# v2-router-docs-summary-07-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..2054dda --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..a360ce2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,95 @@ +# v2-router-docs-summary-08-health-statuses + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..0ca5792 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..7ec0fc0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,97 @@ +# v2-router-docs-summary-09-manual-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..78ed8cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..a1bdb67 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,89 @@ +# v2-router-docs-summary-10-telegram-integration + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..e9cf5a9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..c53bee3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,93 @@ +# v2-router-find-files-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..b100875 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..00d0a8b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,91 @@ +# v2-router-find-files-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..2e1f88e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..7d725f4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,95 @@ +# v2-router-find-files-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/actions/{action}", + "находится", + "/actions/{action" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..6bcb3fe --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..67a4971 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,97 @@ +# v2-router-find-files-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..22e0dc9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..e5fb3cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,89 @@ +# v2-router-find-files-05-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..0274ae5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..e9e5561 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,93 @@ +# v2-router-find-files-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..f395d9e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..06d4c48 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,97 @@ +# v2-router-find-files-07-errors-catalog + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "errors_catalog", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..f9b1b13 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..fa65c6a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,91 @@ +# v2-router-find-files-08-docs-index + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "docs_index", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..083049f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..e633f25 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,95 @@ +# v2-router-find-files-09-telegram-bot-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..982e56f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..5f77518 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,95 @@ +# v2-router-find-files-10-telegram-notify-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каких", + "файлах", + "описан", + "worker", + "telegram_notify" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "telegram_notify" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..27d18f8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..aa7263c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,79 @@ +# v2-router-general-01-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..4480103 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..d662352 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,89 @@ +# v2-router-general-02-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "test_echo_app" + ], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..d6694a2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..e0055a8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,85 @@ +# v2-router-general-03-problem + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..cf9f2ed --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..adba625 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,83 @@ +# v2-router-general-04-docs-contents + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..bb52214 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..4dd636a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,85 @@ +# v2-router-general-05-main-parts + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..b5a6df4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..693f1cf --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,83 @@ +# v2-router-general-06-high-level-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..1336596 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..72b9d53 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,89 @@ +# v2-router-general-07-besides-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "тут", + "есть", + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..e7aa4e4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,89 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..8973686 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,99 @@ +# v2-router-general-08-components + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Из чего состоит Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..0b62adf --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..a246294 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,83 @@ +# v2-router-general-09-reading-order + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..2b70a92 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..2df755e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,81 @@ +# v2-router-general-10-short-summary + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/summary.md new file mode 100644 index 0000000..966c540 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_compat/20260407_125007/summary.md @@ -0,0 +1,36 @@ +# pipeline_setup_v4 summary + +Passed: 30/30 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v2-router-general-01-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-02-purpose | process_v2_intent_router | Для чего нужен test_echo_app? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-03-problem | process_v2_intent_router | Какую задачу решает это приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-04-docs-contents | process_v2_intent_router | Что входит в документацию этого проекта? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-05-main-parts | process_v2_intent_router | Какие основные части есть у системы? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-06-high-level-architecture | process_v2_intent_router | Как в целом устроено приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-07-besides-api | process_v2_intent_router | Что тут есть кроме API? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-08-components | process_v2_intent_router | Из чего состоит Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-general-09-reading-order | process_v2_intent_router | Какие документы стоит читать сначала? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-10-short-summary | process_v2_intent_router | Дай короткое summary по проектной документации | GENERAL_QA | SUMMARY | ✓ | +| v2-router-docs-summary-01-health | process_v2_intent_router | Кратко объясни по документации, как работает `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-02-send | process_v2_intent_router | Что делает endpoint `/send`? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-03-actions | process_v2_intent_router | Объясни назначение `/actions/{action}` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-04-runtime-health | process_v2_intent_router | Что такое runtime health в этой документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-05-notification-loop | process_v2_intent_router | Кратко опиши цикл отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-06-architecture | process_v2_intent_router | Как устроена архитектура Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-07-worker | process_v2_intent_router | Что делает worker в этом приложении? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-08-health-statuses | process_v2_intent_router | Какие статусы здоровья есть у runtime? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-09-manual-send | process_v2_intent_router | Как в приложении устроена ручная отправка сообщения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-10-telegram-integration | process_v2_intent_router | Как сервис взаимодействует с Telegram в целом? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-find-files-01-health | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-02-send | process_v2_intent_router | Покажи файл с описанием endpoint `/send` | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-03-actions | process_v2_intent_router | Где находится документация по `/actions/{action}`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-04-runtime-health | process_v2_intent_router | В каком документе описан runtime health? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-05-loop | process_v2_intent_router | Найди файл с логикой цикла отправки уведомлений | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-06-architecture | process_v2_intent_router | Где описана архитектура приложения? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-07-errors-catalog | process_v2_intent_router | В каком файле лежит каталог ошибок? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-08-docs-index | process_v2_intent_router | Покажи файл-индекс документации проекта | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-09-telegram-bot-api | process_v2_intent_router | Какие файлы относятся к Telegram Bot API интеграции? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-10-telegram-notify-worker | process_v2_intent_router | В каких файлах описан worker `telegram_notify`? | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..7c989bf --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..6d17fdf --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,87 @@ +# v2-router-docs-summary-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..4f9dfb9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..83b7d54 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,87 @@ +# v2-router-docs-summary-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..8612208 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/actions/{action}", + "назначение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..742b36f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,93 @@ +# v2-router-docs-summary-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/actions/{action}", + "назначение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..7ddb660 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..e15eace --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,91 @@ +# v2-router-docs-summary-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..ad7f0c4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..c1d9844 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,93 @@ +# v2-router-docs-summary-05-notification-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notify-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..7821918 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,93 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..18d5da5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,103 @@ +# v2-router-docs-summary-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "устроена", + "архитектура", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..55a9191 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..421bafd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,87 @@ +# v2-router-docs-summary-07-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..dc2e465 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "статусы", + "здоровья", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "статусы", + "здоровья", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..bfcf2cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,91 @@ +# v2-router-docs-summary-08-health-statuses + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "статусы", + "здоровья", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "статусы", + "здоровья", + "runtime" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..bc2ffd0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..b658419 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,95 @@ +# v2-router-docs-summary-09-manual-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..133f00c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..77df245 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,89 @@ +# v2-router-docs-summary-10-telegram-integration + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис", + "взаимодействует", + "telegram", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..4353690 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..cc44e90 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,93 @@ +# v2-router-find-files-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..fbb207f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/send", + "описанием" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..6b3c64a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,89 @@ +# v2-router-find-files-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/send", + "описанием" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..0eb1e76 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/actions/{action}", + "находится" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..62a7292 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,93 @@ +# v2-router-find-files-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/actions/{action}", + "находится" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..40de3ef --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..1dfe736 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,95 @@ +# v2-router-find-files-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..bf38dab --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..47a101f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,89 @@ +# v2-router-find-files-05-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..48e96de --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..d2e2182 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,91 @@ +# v2-router-find-files-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..7e02ad0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..9cb49d4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,95 @@ +# v2-router-find-files-07-errors-catalog + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "errors_catalog" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..30ab76f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..bd52df0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,89 @@ +# v2-router-find-files-08-docs-index + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs_index" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..10f692c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..bc25ac2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,95 @@ +# v2-router-find-files-09-telegram-bot-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "относятся", + "telegram", + "bot", + "api", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..901cd5d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..72a7da6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,91 @@ +# v2-router-find-files-10-telegram-notify-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..2a7445a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..4b09b55 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,79 @@ +# v2-router-general-01-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..1d8f479 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "test_echo_app", + "чего", + "нужен" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "test_echo_app", + "чего", + "нужен" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..3bb6f54 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,83 @@ +# v2-router-general-02-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "test_echo_app", + "чего", + "нужен" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "test_echo_app", + "чего", + "нужен" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..015580b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..c7c182d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,85 @@ +# v2-router-general-03-problem + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "какую", + "задачу", + "решает", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..530abcb --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..c2fbaf0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,83 @@ +# v2-router-general-04-docs-contents + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "входит", + "документацию", + "проекта" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..c19325f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "основные", + "части", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..dd6e2ef --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,83 @@ +# v2-router-general-05-main-parts + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "основные", + "части", + "системы" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..ec496b3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..c72e3f8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,83 @@ +# v2-router-general-06-high-level-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..edb215e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..de030b2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,85 @@ +# v2-router-general-07-besides-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "кроме", + "api" + ], + "anchors": { + "entity_names": [ + "API" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..e7aa4e4 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,89 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..8973686 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,99 @@ +# v2-router-general-08-components + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Из чего состоит Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "чего", + "состоит", + "telegram", + "notify", + "app" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..ecd9d97 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..48cdf2e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,83 @@ +# v2-router-general-09-reading-order + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..ab2692c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..c874c73 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,81 @@ +# v2-router-general-10-short-summary + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "короткое", + "проектной" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/summary.md new file mode 100644 index 0000000..966c540 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_postfix_compat/20260407_125827/summary.md @@ -0,0 +1,36 @@ +# pipeline_setup_v4 summary + +Passed: 30/30 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v2-router-general-01-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-02-purpose | process_v2_intent_router | Для чего нужен test_echo_app? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-03-problem | process_v2_intent_router | Какую задачу решает это приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-04-docs-contents | process_v2_intent_router | Что входит в документацию этого проекта? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-05-main-parts | process_v2_intent_router | Какие основные части есть у системы? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-06-high-level-architecture | process_v2_intent_router | Как в целом устроено приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-07-besides-api | process_v2_intent_router | Что тут есть кроме API? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-08-components | process_v2_intent_router | Из чего состоит Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-general-09-reading-order | process_v2_intent_router | Какие документы стоит читать сначала? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-10-short-summary | process_v2_intent_router | Дай короткое summary по проектной документации | GENERAL_QA | SUMMARY | ✓ | +| v2-router-docs-summary-01-health | process_v2_intent_router | Кратко объясни по документации, как работает `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-02-send | process_v2_intent_router | Что делает endpoint `/send`? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-03-actions | process_v2_intent_router | Объясни назначение `/actions/{action}` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-04-runtime-health | process_v2_intent_router | Что такое runtime health в этой документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-05-notification-loop | process_v2_intent_router | Кратко опиши цикл отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-06-architecture | process_v2_intent_router | Как устроена архитектура Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-07-worker | process_v2_intent_router | Что делает worker в этом приложении? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-08-health-statuses | process_v2_intent_router | Какие статусы здоровья есть у runtime? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-09-manual-send | process_v2_intent_router | Как в приложении устроена ручная отправка сообщения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-10-telegram-integration | process_v2_intent_router | Как сервис взаимодействует с Telegram в целом? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-find-files-01-health | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-02-send | process_v2_intent_router | Покажи файл с описанием endpoint `/send` | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-03-actions | process_v2_intent_router | Где находится документация по `/actions/{action}`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-04-runtime-health | process_v2_intent_router | В каком документе описан runtime health? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-05-loop | process_v2_intent_router | Найди файл с логикой цикла отправки уведомлений | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-06-architecture | process_v2_intent_router | Где описана архитектура приложения? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-07-errors-catalog | process_v2_intent_router | В каком файле лежит каталог ошибок? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-08-docs-index | process_v2_intent_router | Покажи файл-индекс документации проекта | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-09-telegram-bot-api | process_v2_intent_router | Какие файлы относятся к Telegram Bot API интеграции? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-10-telegram-notify-worker | process_v2_intent_router | В каких файлах описан worker `telegram_notify`? | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-01-endpoint.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-01-endpoint.json new file mode 100644 index 0000000..74dc15a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-01-endpoint.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-docs-summary-01-endpoint", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-01-endpoint.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-01-endpoint.md new file mode 100644 index 0000000..2a190d5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-01-endpoint.md @@ -0,0 +1,87 @@ +# v3-docs-summary-01-endpoint + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Объясни по документации endpoint `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-02-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-02-architecture.json new file mode 100644 index 0000000..2a57176 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-02-architecture.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-02-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-02-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-02-architecture.md new file mode 100644 index 0000000..19bd099 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-02-architecture.md @@ -0,0 +1,93 @@ +# v3-docs-summary-02-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как устроена архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-03-process.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-03-process.json new file mode 100644 index 0000000..fe982bd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-03-process.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-docs-summary-03-process", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-03-process.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-03-process.md new file mode 100644 index 0000000..cb54efa --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-03-process.md @@ -0,0 +1,85 @@ +# v3-docs-summary-03-process + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Опиши процесс отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-04-entity.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-04-entity.json new file mode 100644 index 0000000..e6948f5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-04-entity.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-04-entity", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-04-entity.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-04-entity.md new file mode 100644 index 0000000..aa95a4b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-docs-summary-04-entity.md @@ -0,0 +1,93 @@ +# v3-docs-summary-04-entity + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что такое runtime health в документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-01-which-file.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-01-which-file.json new file mode 100644 index 0000000..1473439 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-01-which-file.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-01-which-file", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-01-which-file.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-01-which-file.md new file mode 100644 index 0000000..c3e030c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-01-which-file.md @@ -0,0 +1,93 @@ +# v3-find-files-01-which-file + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-02-show-doc.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-02-show-doc.json new file mode 100644 index 0000000..774eadc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-02-show-doc.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-02-show-doc", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-02-show-doc.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-02-show-doc.md new file mode 100644 index 0000000..12d9de3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-02-show-doc.md @@ -0,0 +1,93 @@ +# v3-find-files-02-show-doc + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Покажи документ про runtime health + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-03-where-described.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-03-where-described.json new file mode 100644 index 0000000..2680d84 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-03-where-described.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-03-where-described", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-03-where-described.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-03-where-described.md new file mode 100644 index 0000000..5074f19 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-find-files-03-where-described.md @@ -0,0 +1,93 @@ +# v3-find-files-03-where-described + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где описано `docs/architecture/overview.md`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-01-overview.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-01-overview.json new file mode 100644 index 0000000..6c2877d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-01-overview.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-01-overview", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-01-overview.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-01-overview.md new file mode 100644 index 0000000..1a8372c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-01-overview.md @@ -0,0 +1,83 @@ +# v3-general-01-overview + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Общий обзор сервиса + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-02-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-02-what-is-service.json new file mode 100644 index 0000000..26d33e6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-02-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-02-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-02-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-02-what-is-service.md new file mode 100644 index 0000000..1f3ce10 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-02-what-is-service.md @@ -0,0 +1,79 @@ +# v3-general-02-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-03-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-03-purpose.json new file mode 100644 index 0000000..9cbced6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-03-purpose.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-03-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-03-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-03-purpose.md new file mode 100644 index 0000000..d21bc3e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-general-03-purpose.md @@ -0,0 +1,83 @@ +# v3-general-03-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Для чего нужен этот сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-01-operational.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-01-operational.json new file mode 100644 index 0000000..79e9d03 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-01-operational.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-negative-01-operational", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-01-operational.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-01-operational.md new file mode 100644 index 0000000..cb7720c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-01-operational.md @@ -0,0 +1,87 @@ +# v3-negative-01-operational + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Почему у меня сейчас упал деплой? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-02-meta.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-02-meta.json new file mode 100644 index 0000000..f1145a1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-02-meta.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v3-negative-02-meta", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-02-meta.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-02-meta.md new file mode 100644 index 0000000..ed21532 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-02-meta.md @@ -0,0 +1,81 @@ +# v3-negative-02-meta + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как ты выбираешь маршрут? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-03-non-docs.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-03-non-docs.json new file mode 100644 index 0000000..cf49152 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-03-non-docs.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-negative-03-non-docs", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-03-non-docs.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-03-non-docs.md new file mode 100644 index 0000000..6bf263d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-negative-03-non-docs.md @@ -0,0 +1,85 @@ +# v3-negative-03-non-docs + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Сколько сейчас времени на сервере? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-01-colloquial.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-01-colloquial.json new file mode 100644 index 0000000..adacddb --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-01-colloquial.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v3-noisy-01-colloquial", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-01-colloquial.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-01-colloquial.md new file mode 100644 index 0000000..cc51cf6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-01-colloquial.md @@ -0,0 +1,97 @@ +# v3-noisy-01-colloquial + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где там дока про health, покажи плз + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-02-abbrev.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-02-abbrev.json new file mode 100644 index 0000000..f2c1731 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-02-abbrev.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v3-noisy-02-abbrev", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-02-abbrev.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-02-abbrev.md new file mode 100644 index 0000000..4364012 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-02-abbrev.md @@ -0,0 +1,95 @@ +# v3-noisy-02-abbrev + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Нужен краткий док-саммари по api /send + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-03-broken-phrase.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-03-broken-phrase.json new file mode 100644 index 0000000..88082c2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-03-broken-phrase.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v3-noisy-03-broken-phrase", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-03-broken-phrase.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-03-broken-phrase.md new file mode 100644 index 0000000..24577cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/router_llm_first_v3_v3-noisy-03-broken-phrase.md @@ -0,0 +1,95 @@ +# v3-noisy-03-broken-phrase + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +runtime health это где описано в доках + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/summary.md new file mode 100644 index 0000000..fee2567 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_postfix/20260407_125805/summary.md @@ -0,0 +1,22 @@ +# pipeline_setup_v4 summary + +Passed: 16/16 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v3-docs-summary-01-endpoint | process_v2_intent_router | Объясни по документации endpoint `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-02-architecture | process_v2_intent_router | Как устроена архитектура приложения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-03-process | process_v2_intent_router | Опиши процесс отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-04-entity | process_v2_intent_router | Что такое runtime health в документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-find-files-01-which-file | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-02-show-doc | process_v2_intent_router | Покажи документ про runtime health | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-03-where-described | process_v2_intent_router | Где описано `docs/architecture/overview.md`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-general-01-overview | process_v2_intent_router | Общий обзор сервиса | GENERAL_QA | SUMMARY | ✓ | +| v3-general-02-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-general-03-purpose | process_v2_intent_router | Для чего нужен этот сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-01-operational | process_v2_intent_router | Почему у меня сейчас упал деплой? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-02-meta | process_v2_intent_router | Как ты выбираешь маршрут? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-03-non-docs | process_v2_intent_router | Сколько сейчас времени на сервере? | GENERAL_QA | SUMMARY | ✓ | +| v3-noisy-01-colloquial | process_v2_intent_router | Где там дока про health, покажи плз | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-noisy-02-abbrev | process_v2_intent_router | Нужен краткий док-саммари по api /send | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-noisy-03-broken-phrase | process_v2_intent_router | runtime health это где описано в доках | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-01-endpoint.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-01-endpoint.json new file mode 100644 index 0000000..19e26c7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-01-endpoint.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v3-docs-summary-01-endpoint", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-01-endpoint.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-01-endpoint.md new file mode 100644 index 0000000..c39757f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-01-endpoint.md @@ -0,0 +1,89 @@ +# v3-docs-summary-01-endpoint + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Объясни по документации endpoint `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-02-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-02-architecture.json new file mode 100644 index 0000000..2a57176 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-02-architecture.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-02-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-02-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-02-architecture.md new file mode 100644 index 0000000..19bd099 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-02-architecture.md @@ -0,0 +1,93 @@ +# v3-docs-summary-02-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как устроена архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-03-process.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-03-process.json new file mode 100644 index 0000000..01283be --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-03-process.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-docs-summary-03-process", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-03-process.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-03-process.md new file mode 100644 index 0000000..c7439dd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-03-process.md @@ -0,0 +1,85 @@ +# v3-docs-summary-03-process + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Опиши процесс отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-04-entity.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-04-entity.json new file mode 100644 index 0000000..e6948f5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-04-entity.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-04-entity", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-04-entity.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-04-entity.md new file mode 100644 index 0000000..aa95a4b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-docs-summary-04-entity.md @@ -0,0 +1,93 @@ +# v3-docs-summary-04-entity + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что такое runtime health в документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-01-which-file.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-01-which-file.json new file mode 100644 index 0000000..9256884 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-01-which-file.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-01-which-file", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-01-which-file.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-01-which-file.md new file mode 100644 index 0000000..6ae9877 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-01-which-file.md @@ -0,0 +1,93 @@ +# v3-find-files-01-which-file + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-02-show-doc.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-02-show-doc.json new file mode 100644 index 0000000..77c7f5b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-02-show-doc.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-02-show-doc", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-02-show-doc.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-02-show-doc.md new file mode 100644 index 0000000..50b93d6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-02-show-doc.md @@ -0,0 +1,93 @@ +# v3-find-files-02-show-doc + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Покажи документ про runtime health + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-03-where-described.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-03-where-described.json new file mode 100644 index 0000000..0e5e9f8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-03-where-described.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-03-where-described", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-03-where-described.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-03-where-described.md new file mode 100644 index 0000000..81a1a8f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-find-files-03-where-described.md @@ -0,0 +1,93 @@ +# v3-find-files-03-where-described + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где описано `docs/architecture/overview.md`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-01-overview.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-01-overview.json new file mode 100644 index 0000000..9659127 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-01-overview.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-01-overview", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-01-overview.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-01-overview.md new file mode 100644 index 0000000..795dcdb --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-01-overview.md @@ -0,0 +1,83 @@ +# v3-general-01-overview + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Общий обзор сервиса + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-02-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-02-what-is-service.json new file mode 100644 index 0000000..6f22b47 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-02-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-02-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-02-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-02-what-is-service.md new file mode 100644 index 0000000..c2e4650 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-02-what-is-service.md @@ -0,0 +1,79 @@ +# v3-general-02-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-03-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-03-purpose.json new file mode 100644 index 0000000..c9e2a55 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-03-purpose.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-general-03-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-03-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-03-purpose.md new file mode 100644 index 0000000..fc18f93 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-general-03-purpose.md @@ -0,0 +1,85 @@ +# v3-general-03-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Для чего нужен этот сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-01-operational.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-01-operational.json new file mode 100644 index 0000000..dfa4498 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-01-operational.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-negative-01-operational", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-01-operational.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-01-operational.md new file mode 100644 index 0000000..0d5d63e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-01-operational.md @@ -0,0 +1,87 @@ +# v3-negative-01-operational + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Почему у меня сейчас упал деплой? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-02-meta.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-02-meta.json new file mode 100644 index 0000000..20b8c25 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-02-meta.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v3-negative-02-meta", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-02-meta.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-02-meta.md new file mode 100644 index 0000000..5954af9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-02-meta.md @@ -0,0 +1,81 @@ +# v3-negative-02-meta + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как ты выбираешь маршрут? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-03-non-docs.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-03-non-docs.json new file mode 100644 index 0000000..92f2143 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-03-non-docs.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-negative-03-non-docs", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-03-non-docs.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-03-non-docs.md new file mode 100644 index 0000000..6c37d74 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-negative-03-non-docs.md @@ -0,0 +1,85 @@ +# v3-negative-03-non-docs + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Сколько сейчас времени на сервере? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-01-colloquial.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-01-colloquial.json new file mode 100644 index 0000000..f8d7d4f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-01-colloquial.json @@ -0,0 +1,89 @@ +{ + "meta": { + "case_id": "v3-noisy-01-colloquial", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": false, + "mismatches": [ + "sub_intent: expected FIND_FILES, got SUMMARY" + ] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-01-colloquial.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-01-colloquial.md new file mode 100644 index 0000000..c5ae023 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-01-colloquial.md @@ -0,0 +1,97 @@ +# v3-noisy-01-colloquial + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: False + +## Query +Где там дока про health, покажи плз + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- sub_intent: expected FIND_FILES, got SUMMARY \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-02-abbrev.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-02-abbrev.json new file mode 100644 index 0000000..aa38be7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-02-abbrev.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v3-noisy-02-abbrev", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-02-abbrev.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-02-abbrev.md new file mode 100644 index 0000000..c9b24c5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-02-abbrev.md @@ -0,0 +1,95 @@ +# v3-noisy-02-abbrev + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Нужен краткий док-саммари по api /send + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-03-broken-phrase.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-03-broken-phrase.json new file mode 100644 index 0000000..14503a3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-03-broken-phrase.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v3-noisy-03-broken-phrase", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-03-broken-phrase.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-03-broken-phrase.md new file mode 100644 index 0000000..e4c3e72 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/router_llm_first_v3_v3-noisy-03-broken-phrase.md @@ -0,0 +1,95 @@ +# v3-noisy-03-broken-phrase + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +runtime health это где описано в доках + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/summary.md new file mode 100644 index 0000000..f329ebf --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124942/summary.md @@ -0,0 +1,25 @@ +# pipeline_setup_v4 summary + +Passed: 15/16 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v3-docs-summary-01-endpoint | process_v2_intent_router | Объясни по документации endpoint `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-02-architecture | process_v2_intent_router | Как устроена архитектура приложения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-03-process | process_v2_intent_router | Опиши процесс отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-04-entity | process_v2_intent_router | Что такое runtime health в документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-find-files-01-which-file | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-02-show-doc | process_v2_intent_router | Покажи документ про runtime health | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-03-where-described | process_v2_intent_router | Где описано `docs/architecture/overview.md`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-general-01-overview | process_v2_intent_router | Общий обзор сервиса | GENERAL_QA | SUMMARY | ✓ | +| v3-general-02-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-general-03-purpose | process_v2_intent_router | Для чего нужен этот сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-01-operational | process_v2_intent_router | Почему у меня сейчас упал деплой? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-02-meta | process_v2_intent_router | Как ты выбираешь маршрут? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-03-non-docs | process_v2_intent_router | Сколько сейчас времени на сервере? | GENERAL_QA | SUMMARY | ✓ | +| v3-noisy-01-colloquial | process_v2_intent_router | Где там дока про health, покажи плз | DOC_EXPLAIN | SUMMARY | ✗ | +| v3-noisy-02-abbrev | process_v2_intent_router | Нужен краткий док-саммари по api /send | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-noisy-03-broken-phrase | process_v2_intent_router | runtime health это где описано в доках | DOC_EXPLAIN | FIND_FILES | ✓ | + +## Failures +- **v3-noisy-01-colloquial**: sub_intent: expected FIND_FILES, got SUMMARY \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-01-endpoint.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-01-endpoint.json new file mode 100644 index 0000000..19e26c7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-01-endpoint.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v3-docs-summary-01-endpoint", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-01-endpoint.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-01-endpoint.md new file mode 100644 index 0000000..c39757f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-01-endpoint.md @@ -0,0 +1,89 @@ +# v3-docs-summary-01-endpoint + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Объясни по документации endpoint `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health", + "endpoint" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-02-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-02-architecture.json new file mode 100644 index 0000000..2a57176 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-02-architecture.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-02-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-02-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-02-architecture.md new file mode 100644 index 0000000..19bd099 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-02-architecture.md @@ -0,0 +1,93 @@ +# v3-docs-summary-02-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как устроена архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-03-process.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-03-process.json new file mode 100644 index 0000000..01283be --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-03-process.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-docs-summary-03-process", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-03-process.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-03-process.md new file mode 100644 index 0000000..c7439dd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-03-process.md @@ -0,0 +1,85 @@ +# v3-docs-summary-03-process + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Опиши процесс отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-04-entity.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-04-entity.json new file mode 100644 index 0000000..e6948f5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-04-entity.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-04-entity", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-04-entity.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-04-entity.md new file mode 100644 index 0000000..aa95a4b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-docs-summary-04-entity.md @@ -0,0 +1,93 @@ +# v3-docs-summary-04-entity + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что такое runtime health в документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "runtime_health", + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-01-which-file.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-01-which-file.json new file mode 100644 index 0000000..9256884 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-01-which-file.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-01-which-file", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-01-which-file.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-01-which-file.md new file mode 100644 index 0000000..6ae9877 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-01-which-file.md @@ -0,0 +1,93 @@ +# v3-find-files-01-which-file + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-02-show-doc.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-02-show-doc.json new file mode 100644 index 0000000..77c7f5b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-02-show-doc.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-02-show-doc", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-02-show-doc.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-02-show-doc.md new file mode 100644 index 0000000..50b93d6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-02-show-doc.md @@ -0,0 +1,93 @@ +# v3-find-files-02-show-doc + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Покажи документ про runtime health + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-03-where-described.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-03-where-described.json new file mode 100644 index 0000000..0e5e9f8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-03-where-described.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-03-where-described", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-03-where-described.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-03-where-described.md new file mode 100644 index 0000000..81a1a8f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-find-files-03-where-described.md @@ -0,0 +1,93 @@ +# v3-find-files-03-where-described + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где описано `docs/architecture/overview.md`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-01-overview.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-01-overview.json new file mode 100644 index 0000000..9659127 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-01-overview.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-01-overview", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-01-overview.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-01-overview.md new file mode 100644 index 0000000..795dcdb --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-01-overview.md @@ -0,0 +1,83 @@ +# v3-general-01-overview + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Общий обзор сервиса + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-02-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-02-what-is-service.json new file mode 100644 index 0000000..6f22b47 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-02-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-02-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-02-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-02-what-is-service.md new file mode 100644 index 0000000..c2e4650 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-02-what-is-service.md @@ -0,0 +1,79 @@ +# v3-general-02-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-03-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-03-purpose.json new file mode 100644 index 0000000..c9e2a55 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-03-purpose.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-general-03-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-03-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-03-purpose.md new file mode 100644 index 0000000..fc18f93 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-general-03-purpose.md @@ -0,0 +1,85 @@ +# v3-general-03-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Для чего нужен этот сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "для", + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-01-operational.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-01-operational.json new file mode 100644 index 0000000..dfa4498 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-01-operational.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-negative-01-operational", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-01-operational.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-01-operational.md new file mode 100644 index 0000000..0d5d63e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-01-operational.md @@ -0,0 +1,87 @@ +# v3-negative-01-operational + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Почему у меня сейчас упал деплой? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-02-meta.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-02-meta.json new file mode 100644 index 0000000..20b8c25 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-02-meta.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v3-negative-02-meta", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-02-meta.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-02-meta.md new file mode 100644 index 0000000..5954af9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-02-meta.md @@ -0,0 +1,81 @@ +# v3-negative-02-meta + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как ты выбираешь маршрут? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-03-non-docs.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-03-non-docs.json new file mode 100644 index 0000000..92f2143 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-03-non-docs.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-negative-03-non-docs", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-03-non-docs.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-03-non-docs.md new file mode 100644 index 0000000..6c37d74 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-negative-03-non-docs.md @@ -0,0 +1,85 @@ +# v3-negative-03-non-docs + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Сколько сейчас времени на сервере? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-01-colloquial.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-01-colloquial.json new file mode 100644 index 0000000..02660ca --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-01-colloquial.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v3-noisy-01-colloquial", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-01-colloquial.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-01-colloquial.md new file mode 100644 index 0000000..6141922 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-01-colloquial.md @@ -0,0 +1,97 @@ +# v3-noisy-01-colloquial + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где там дока про health, покажи плз + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-02-abbrev.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-02-abbrev.json new file mode 100644 index 0000000..aa38be7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-02-abbrev.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v3-noisy-02-abbrev", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-02-abbrev.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-02-abbrev.md new file mode 100644 index 0000000..c9b24c5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-02-abbrev.md @@ -0,0 +1,95 @@ +# v3-noisy-02-abbrev + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Нужен краткий док-саммари по api /send + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-03-broken-phrase.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-03-broken-phrase.json new file mode 100644 index 0000000..14503a3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-03-broken-phrase.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v3-noisy-03-broken-phrase", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-03-broken-phrase.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-03-broken-phrase.md new file mode 100644 index 0000000..e4c3e72 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/router_llm_first_v3_v3-noisy-03-broken-phrase.md @@ -0,0 +1,95 @@ +# v3-noisy-03-broken-phrase + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +runtime health это где описано в доках + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime_health", + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/summary.md new file mode 100644 index 0000000..fee2567 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/llm_first_v3_smoke/20260407_124956/summary.md @@ -0,0 +1,22 @@ +# pipeline_setup_v4 summary + +Passed: 16/16 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v3-docs-summary-01-endpoint | process_v2_intent_router | Объясни по документации endpoint `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-02-architecture | process_v2_intent_router | Как устроена архитектура приложения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-03-process | process_v2_intent_router | Опиши процесс отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-04-entity | process_v2_intent_router | Что такое runtime health в документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-find-files-01-which-file | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-02-show-doc | process_v2_intent_router | Покажи документ про runtime health | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-03-where-described | process_v2_intent_router | Где описано `docs/architecture/overview.md`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-general-01-overview | process_v2_intent_router | Общий обзор сервиса | GENERAL_QA | SUMMARY | ✓ | +| v3-general-02-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-general-03-purpose | process_v2_intent_router | Для чего нужен этот сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-01-operational | process_v2_intent_router | Почему у меня сейчас упал деплой? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-02-meta | process_v2_intent_router | Как ты выбираешь маршрут? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-03-non-docs | process_v2_intent_router | Сколько сейчас времени на сервере? | GENERAL_QA | SUMMARY | ✓ | +| v3-noisy-01-colloquial | process_v2_intent_router | Где там дока про health, покажи плз | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-noisy-02-abbrev | process_v2_intent_router | Нужен краткий док-саммари по api /send | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-noisy-03-broken-phrase | process_v2_intent_router | runtime health это где описано в доках | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..a474862 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..3da6762 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..0e631d2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..84c294e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..163fe3e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..f40b533 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..c3e16d3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..dc1d626 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,113 @@ +# v2-router-docs-summary-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..039b19b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..900a1ad --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,113 @@ +# v2-router-docs-summary-05-notification-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..18d10be --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,115 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..da255cd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,125 @@ +# v2-router-docs-summary-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..8720862 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..64e5208 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,105 @@ +# v2-router-docs-summary-07-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..00bfebe --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..2aef934 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-08-health-statuses + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..ea42750 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,107 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..c808ec2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,117 @@ +# v2-router-docs-summary-09-manual-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..52fc26c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,98 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": false, + "mismatches": [ + "domain: expected DOCS, got GENERAL", + "intent: expected DOC_EXPLAIN, got GENERAL_QA" + ] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_fallback", + "llm_router_used": false, + "confidence": 0.0 + }, + "details": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.0, + "routing_mode": "llm_fallback", + "llm_router_used": false, + "reason_short": "fallback to GENERAL_QA.SUMMARY" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "fallback to GENERAL_QA.SUMMARY", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..ab810ba --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,106 @@ +# v2-router-docs-summary-10-telegram-integration + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: False + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_fallback", + "llm_router_used": false, + "confidence": 0.0 +} +``` + +## Details +```json +{ + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.0, + "routing_mode": "llm_fallback", + "llm_router_used": false, + "reason_short": "fallback to GENERAL_QA.SUMMARY" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "fallback to GENERAL_QA.SUMMARY", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- domain: expected DOCS, got GENERAL +- intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..c5dbf06 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..9870147 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,109 @@ +# v2-router-find-files-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..1a2c1aa --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..47e2080 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,105 @@ +# v2-router-find-files-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..1204332 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..e25674d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,109 @@ +# v2-router-find-files-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..7bccd5a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..ad030e5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,113 @@ +# v2-router-find-files-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..d05481c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..6736e8f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,101 @@ +# v2-router-find-files-05-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..8dd2948 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..4411a1a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,105 @@ +# v2-router-find-files-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..7976947 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..8175992 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,113 @@ +# v2-router-find-files-07-errors-catalog + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..c1c5b7c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..1946d5c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,101 @@ +# v2-router-find-files-08-docs-index + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..42be420 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..2f1b0c1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,109 @@ +# v2-router-find-files-09-telegram-bot-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..4b2866b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..c0b6454 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,105 @@ +# v2-router-find-files-10-telegram-notify-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..9eef4cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..e007ea0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,93 @@ +# v2-router-general-01-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..9aed4f1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..9bcf726 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,97 @@ +# v2-router-general-02-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..0d45adc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..1a3c320 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,101 @@ +# v2-router-general-03-problem + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..f57afcc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..cfcb6e8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,101 @@ +# v2-router-general-04-docs-contents + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..c697753 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..a3ddb38 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,97 @@ +# v2-router-general-05-main-parts + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..16bab4d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..06d558a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,97 @@ +# v2-router-general-06-high-level-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..f451d58 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..cd93ca7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,105 @@ +# v2-router-general-07-besides-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..47390ed --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,106 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": false, + "mismatches": [ + "domain: expected GENERAL, got DOCS", + "intent: expected GENERAL_QA, got DOC_EXPLAIN" + ] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..c0416ea --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,114 @@ +# v2-router-general-08-components + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: False + +## Query +Из чего состоит Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- domain: expected GENERAL, got DOCS +- intent: expected GENERAL_QA, got DOC_EXPLAIN \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..cca6d29 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..02662d1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,93 @@ +# v2-router-general-09-reading-order + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..ce8e869 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..be3dbe5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,97 @@ +# v2-router-general-10-short-summary + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/summary.md new file mode 100644 index 0000000..9a8e5ed --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4/20260407_120038/summary.md @@ -0,0 +1,40 @@ +# pipeline_setup_v4 summary + +Passed: 28/30 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v2-router-general-01-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-02-purpose | process_v2_intent_router | Для чего нужен test_echo_app? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-03-problem | process_v2_intent_router | Какую задачу решает это приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-04-docs-contents | process_v2_intent_router | Что входит в документацию этого проекта? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-05-main-parts | process_v2_intent_router | Какие основные части есть у системы? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-06-high-level-architecture | process_v2_intent_router | Как в целом устроено приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-07-besides-api | process_v2_intent_router | Что тут есть кроме API? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-08-components | process_v2_intent_router | Из чего состоит Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✗ | +| v2-router-general-09-reading-order | process_v2_intent_router | Какие документы стоит читать сначала? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-10-short-summary | process_v2_intent_router | Дай короткое summary по проектной документации | GENERAL_QA | SUMMARY | ✓ | +| v2-router-docs-summary-01-health | process_v2_intent_router | Кратко объясни по документации, как работает `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-02-send | process_v2_intent_router | Что делает endpoint `/send`? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-03-actions | process_v2_intent_router | Объясни назначение `/actions/{action}` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-04-runtime-health | process_v2_intent_router | Что такое runtime health в этой документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-05-notification-loop | process_v2_intent_router | Кратко опиши цикл отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-06-architecture | process_v2_intent_router | Как устроена архитектура Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-07-worker | process_v2_intent_router | Что делает worker в этом приложении? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-08-health-statuses | process_v2_intent_router | Какие статусы здоровья есть у runtime? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-09-manual-send | process_v2_intent_router | Как в приложении устроена ручная отправка сообщения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-10-telegram-integration | process_v2_intent_router | Как сервис взаимодействует с Telegram в целом? | GENERAL_QA | SUMMARY | ✗ | +| v2-router-find-files-01-health | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-02-send | process_v2_intent_router | Покажи файл с описанием endpoint `/send` | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-03-actions | process_v2_intent_router | Где находится документация по `/actions/{action}`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-04-runtime-health | process_v2_intent_router | В каком документе описан runtime health? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-05-loop | process_v2_intent_router | Найди файл с логикой цикла отправки уведомлений | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-06-architecture | process_v2_intent_router | Где описана архитектура приложения? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-07-errors-catalog | process_v2_intent_router | В каком файле лежит каталог ошибок? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-08-docs-index | process_v2_intent_router | Покажи файл-индекс документации проекта | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-09-telegram-bot-api | process_v2_intent_router | Какие файлы относятся к Telegram Bot API интеграции? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-10-telegram-notify-worker | process_v2_intent_router | В каких файлах описан worker `telegram_notify`? | DOC_EXPLAIN | FIND_FILES | ✓ | + +## Failures +- **v2-router-general-08-components**: domain: expected GENERAL, got DOCS; intent: expected GENERAL_QA, got DOC_EXPLAIN +- **v2-router-docs-summary-10-telegram-integration**: domain: expected DOCS, got GENERAL; intent: expected DOC_EXPLAIN, got GENERAL_QA \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104.zip b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104.zip new file mode 100644 index 0000000..f0be01b Binary files /dev/null and b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104.zip differ diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json new file mode 100644 index 0000000..a474862 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md new file mode 100644 index 0000000..3da6762 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-01-health.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко объясни по документации, как работает `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Кратко объясни по документации, как работает `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко объясни по документации, как работает `/health`", + "normalized_query": "Кратко объясни по документации, как работает `/health`", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко объясни по документации, как работает `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "Кратко", + "как", + "работает" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json new file mode 100644 index 0000000..0e631d2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md new file mode 100644 index 0000000..84c294e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-02-send.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает endpoint `/send`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что делает endpoint `/send`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает endpoint `/send`?", + "normalized_query": "Что делает endpoint `/send`?", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает endpoint `/send`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Что", + "делает", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json new file mode 100644 index 0000000..163fe3e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md new file mode 100644 index 0000000..f40b533 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-03-actions.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Объясни назначение `/actions/{action}` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Объясни назначение `/actions/{action}`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни назначение `/actions/{action}`", + "normalized_query": "Объясни назначение `/actions/{action}`", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни назначение `/actions/{action}`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "назначение", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json new file mode 100644 index 0000000..c3e16d3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md new file mode 100644 index 0000000..dc1d626 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-04-runtime-health.md @@ -0,0 +1,113 @@ +# v2-router-docs-summary-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что такое runtime health в этой документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в этой документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в этой документации?", + "normalized_query": "Что такое runtime health в этой документации?", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в этой документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "Что", + "такое", + "runtime", + "health", + "этой" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json new file mode 100644 index 0000000..039b19b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-05-notification-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md new file mode 100644 index 0000000..900a1ad --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-05-notification-loop.md @@ -0,0 +1,113 @@ +# v2-router-docs-summary-05-notification-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Кратко опиши цикл отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Кратко опиши цикл отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Кратко опиши цикл отправки уведомлений", + "normalized_query": "Кратко опиши цикл отправки уведомлений", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Кратко опиши цикл отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notification-loop", + "Кратко", + "опиши", + "цикл", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [ + "telegram-notification-loop" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json new file mode 100644 index 0000000..18d10be --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.json @@ -0,0 +1,115 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md new file mode 100644 index 0000000..da255cd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-06-architecture.md @@ -0,0 +1,125 @@ +# v2-router-docs-summary-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как устроена архитектура Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура Telegram Notify App?", + "normalized_query": "Как устроена архитектура Telegram Notify App?", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "telegram-notify-app-overview", + "Telegram", + "Notify", + "App", + "Как", + "устроена", + "архитектура" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json new file mode 100644 index 0000000..8720862 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-07-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md new file mode 100644 index 0000000..64e5208 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-07-worker.md @@ -0,0 +1,105 @@ +# v2-router-docs-summary-07-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что делает worker в этом приложении? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что делает worker в этом приложении?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что делает worker в этом приложении?", + "normalized_query": "Что делает worker в этом приложении?", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что делает worker в этом приложении?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "делает", + "worker", + "этом", + "приложении" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json new file mode 100644 index 0000000..00bfebe --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-08-health-statuses", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md new file mode 100644 index 0000000..2aef934 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-08-health-statuses.md @@ -0,0 +1,109 @@ +# v2-router-docs-summary-08-health-statuses + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие статусы здоровья есть у runtime? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие статусы здоровья есть у runtime?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Какие статусы здоровья есть у runtime?", + "normalized_query": "Какие статусы здоровья есть у runtime?", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие статусы здоровья есть у runtime?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "статусы", + "здоровья", + "есть", + "runtime" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json new file mode 100644 index 0000000..ea42750 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.json @@ -0,0 +1,107 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-09-manual-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md new file mode 100644 index 0000000..c808ec2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-09-manual-send.md @@ -0,0 +1,117 @@ +# v2-router-docs-summary-09-manual-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в приложении устроена ручная отправка сообщения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Как в приложении устроена ручная отправка сообщения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как в приложении устроена ручная отправка сообщения?", + "normalized_query": "Как в приложении устроена ручная отправка сообщения?", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в приложении устроена ручная отправка сообщения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "Как", + "приложении", + "устроена", + "ручная", + "отправка", + "сообщения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [ + "/send" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json new file mode 100644 index 0000000..478e2ae --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-docs-summary-10-telegram-integration", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_fallback", + "llm_router_used": false, + "confidence": 0.0 + }, + "details": { + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.0, + "routing_mode": "llm_fallback", + "llm_router_used": false, + "reason_short": "fallback to GENERAL_QA.SUMMARY" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "fallback to GENERAL_QA.SUMMARY", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md new file mode 100644 index 0000000..257b5bd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-docs-summary-10-telegram-integration.md @@ -0,0 +1,105 @@ +# v2-router-docs-summary-10-telegram-integration + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как сервис взаимодействует с Telegram в целом? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_fallback", + "llm_router_used": false, + "confidence": 0.0 +} +``` + +## Details +```json +{ + "query": "Как сервис взаимодействует с Telegram в целом?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как сервис взаимодействует с Telegram в целом?", + "normalized_query": "Как сервис взаимодействует с Telegram в целом?", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.0, + "routing_mode": "llm_fallback", + "llm_router_used": false, + "reason_short": "fallback to GENERAL_QA.SUMMARY" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как сервис взаимодействует с Telegram в целом?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "fallback to GENERAL_QA.SUMMARY", + "target_terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "anchors": { + "entity_names": [ + "Telegram" + ], + "terms": [ + "Telegram", + "Как", + "сервис", + "взаимодействует", + "целом" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-01-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-01-health.json new file mode 100644 index 0000000..c5dbf06 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-01-health.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-find-files-01-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-01-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-01-health.md new file mode 100644 index 0000000..9870147 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-01-health.md @@ -0,0 +1,109 @@ +# v2-router-find-files-01-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "file_names": [ + "/health" + ], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-02-send.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-02-send.json new file mode 100644 index 0000000..1a2c1aa --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-02-send.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-find-files-02-send", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-02-send.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-02-send.md new file mode 100644 index 0000000..47e2080 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-02-send.md @@ -0,0 +1,105 @@ +# v2-router-find-files-02-send + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл с описанием endpoint `/send` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Покажи файл с описанием endpoint `/send`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл с описанием endpoint `/send`", + "normalized_query": "Покажи файл с описанием endpoint `/send`", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл с описанием endpoint `/send`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/send", + "описанием", + "endpoint" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/send", + "описанием", + "endpoint" + ], + "file_names": [ + "/send" + ], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json new file mode 100644 index 0000000..1204332 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-03-actions.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-find-files-03-actions", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md new file mode 100644 index 0000000..e25674d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-03-actions.md @@ -0,0 +1,109 @@ +# v2-router-find-files-03-actions + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где находится документация по `/actions/{action}`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Где находится документация по `/actions/{action}`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где находится документация по `/actions/{action}`?", + "normalized_query": "Где находится документация по `/actions/{action}`?", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где находится документация по `/actions/{action}`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "/actions/{action}", + "находится", + "action" + ], + "anchors": { + "entity_names": [], + "terms": [ + "/actions/{action}", + "находится", + "action" + ], + "file_names": [ + "/actions/{action}" + ], + "endpoint_paths": [ + "/actions/{action}" + ], + "target_doc_hints": [ + "docs/api/control-actions-endpoint.md" + ], + "matched_aliases": [ + "/actions/{action}" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json new file mode 100644 index 0000000..7bccd5a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-find-files-04-runtime-health", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md new file mode 100644 index 0000000..ad030e5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-04-runtime-health.md @@ -0,0 +1,113 @@ +# v2-router-find-files-04-runtime-health + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком документе описан runtime health? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каком документе описан runtime health?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком документе описан runtime health?", + "normalized_query": "В каком документе описан runtime health?", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком документе описан runtime health?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "terms": [ + "runtime-health-entity", + "каком", + "документе", + "описан", + "runtime", + "health" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime-health-entity" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json new file mode 100644 index 0000000..d05481c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-05-loop.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-find-files-05-loop", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md new file mode 100644 index 0000000..6736e8f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-05-loop.md @@ -0,0 +1,101 @@ +# v2-router-find-files-05-loop + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Найди файл с логикой цикла отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Найди файл с логикой цикла отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Найди файл с логикой цикла отправки уведомлений", + "normalized_query": "Найди файл с логикой цикла отправки уведомлений", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Найди файл с логикой цикла отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "terms": [ + "логикой", + "цикла", + "отправки", + "уведомлений" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json new file mode 100644 index 0000000..8dd2948 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-find-files-06-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md new file mode 100644 index 0000000..4411a1a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-06-architecture.md @@ -0,0 +1,105 @@ +# v2-router-find-files-06-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Где описана архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Где описана архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описана архитектура приложения?", + "normalized_query": "Где описана архитектура приложения?", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описана архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram-notify-app-overview", + "описана", + "архитектура", + "приложения" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "telegram-notify-app-overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json new file mode 100644 index 0000000..7976947 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-find-files-07-errors-catalog", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md new file mode 100644 index 0000000..8175992 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-07-errors-catalog.md @@ -0,0 +1,113 @@ +# v2-router-find-files-07-errors-catalog + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каком файле лежит каталог ошибок? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каком файле лежит каталог ошибок?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле лежит каталог ошибок?", + "normalized_query": "В каком файле лежит каталог ошибок?", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле лежит каталог ошибок?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "anchors": { + "entity_names": [], + "terms": [ + "catalog-errors", + "каком", + "файле", + "лежит", + "каталог", + "ошибок" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/errors/catalog.yaml" + ], + "matched_aliases": [ + "catalog-errors" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json new file mode 100644 index 0000000..c1c5b7c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-find-files-08-docs-index", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md new file mode 100644 index 0000000..1946d5c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-08-docs-index.md @@ -0,0 +1,101 @@ +# v2-router-find-files-08-docs-index + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Покажи файл-индекс документации проекта + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Покажи файл-индекс документации проекта", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи файл-индекс документации проекта", + "normalized_query": "Покажи файл-индекс документации проекта", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи файл-индекс документации проекта" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "docs-readme", + "файл-индекс", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/README.md" + ], + "matched_aliases": [ + "docs-readme" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json new file mode 100644 index 0000000..42be420 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.json @@ -0,0 +1,99 @@ +{ + "meta": { + "case_id": "v2-router-find-files-09-telegram-bot-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md new file mode 100644 index 0000000..2f1b0c1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-09-telegram-bot-api.md @@ -0,0 +1,109 @@ +# v2-router-find-files-09-telegram-bot-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие файлы относятся к Telegram Bot API интеграции? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие файлы относятся к Telegram Bot API интеграции?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "normalized_query": "Какие файлы относятся к Telegram Bot API интеграции?", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие файлы относятся к Telegram Bot API интеграции?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Bot", + "API" + ], + "terms": [ + "Telegram", + "Bot", + "API", + "относятся", + "интеграции" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json new file mode 100644 index 0000000..4b2866b --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-find-files-10-telegram-notify-worker", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md new file mode 100644 index 0000000..c0b6454 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-find-files-10-telegram-notify-worker.md @@ -0,0 +1,105 @@ +# v2-router-find-files-10-telegram-notify-worker + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +В каких файлах описан worker `telegram_notify`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "В каких файлах описан worker `telegram_notify`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каких файлах описан worker `telegram_notify`?", + "normalized_query": "В каких файлах описан worker `telegram_notify`?", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic file anchor" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каких файлах описан worker `telegram_notify`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "deterministic file anchor", + "target_terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "anchors": { + "entity_names": [], + "terms": [ + "telegram_notify", + "каких", + "файлах", + "описан", + "worker" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/logic/telegram-notification-loop.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json new file mode 100644 index 0000000..9eef4cc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-general-01-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md new file mode 100644 index 0000000..e007ea0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-01-what-is-service.md @@ -0,0 +1,93 @@ +# v2-router-general-01-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "это", + "сервис" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "это", + "сервис" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-02-purpose.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-02-purpose.json new file mode 100644 index 0000000..9aed4f1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-02-purpose.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-02-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-02-purpose.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-02-purpose.md new file mode 100644 index 0000000..9bcf726 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-02-purpose.md @@ -0,0 +1,97 @@ +# v2-router-general-02-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Для чего нужен test_echo_app? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Для чего нужен test_echo_app?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен test_echo_app?", + "normalized_query": "Для чего нужен test_echo_app?", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен test_echo_app?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Для", + "чего", + "нужен", + "test_echo_app" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-03-problem.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-03-problem.json new file mode 100644 index 0000000..0d45adc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-03-problem.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-general-03-problem", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-03-problem.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-03-problem.md new file mode 100644 index 0000000..1a3c320 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-03-problem.md @@ -0,0 +1,101 @@ +# v2-router-general-03-problem + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какую задачу решает это приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какую задачу решает это приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какую задачу решает это приложение?", + "normalized_query": "Какую задачу решает это приложение?", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какую задачу решает это приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Какую", + "задачу", + "решает", + "это", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json new file mode 100644 index 0000000..f57afcc --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.json @@ -0,0 +1,91 @@ +{ + "meta": { + "case_id": "v2-router-general-04-docs-contents", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md new file mode 100644 index 0000000..cfcb6e8 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-04-docs-contents.md @@ -0,0 +1,101 @@ +# v2-router-general-04-docs-contents + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что входит в документацию этого проекта? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что входит в документацию этого проекта?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что входит в документацию этого проекта?", + "normalized_query": "Что входит в документацию этого проекта?", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что входит в документацию этого проекта?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Что", + "входит", + "документацию", + "этого", + "проекта" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json new file mode 100644 index 0000000..c697753 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-05-main-parts.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-05-main-parts", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md new file mode 100644 index 0000000..a3ddb38 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-05-main-parts.md @@ -0,0 +1,97 @@ +# v2-router-general-05-main-parts + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие основные части есть у системы? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие основные части есть у системы?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие основные части есть у системы?", + "normalized_query": "Какие основные части есть у системы?", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие основные части есть у системы?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "основные", + "части", + "есть", + "системы" + ], + "anchors": { + "entity_names": [], + "terms": [ + "основные", + "части", + "есть", + "системы" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json new file mode 100644 index 0000000..16bab4d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-06-high-level-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md new file mode 100644 index 0000000..06d558a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-06-high-level-architecture.md @@ -0,0 +1,97 @@ +# v2-router-general-06-high-level-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Как в целом устроено приложение? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Как в целом устроено приложение?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как в целом устроено приложение?", + "normalized_query": "Как в целом устроено приложение?", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как в целом устроено приложение?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Как", + "целом", + "устроено", + "приложение" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json new file mode 100644 index 0000000..f451d58 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-07-besides-api.json @@ -0,0 +1,95 @@ +{ + "meta": { + "case_id": "v2-router-general-07-besides-api", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md new file mode 100644 index 0000000..cd93ca7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-07-besides-api.md @@ -0,0 +1,105 @@ +# v2-router-general-07-besides-api + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Что тут есть кроме API? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Что тут есть кроме API?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что тут есть кроме API?", + "normalized_query": "Что тут есть кроме API?", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что тут есть кроме API?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "anchors": { + "entity_names": [ + "API" + ], + "terms": [ + "API", + "Что", + "тут", + "есть", + "кроме" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-08-components.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-08-components.json new file mode 100644 index 0000000..e0bef43 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-08-components.json @@ -0,0 +1,103 @@ +{ + "meta": { + "case_id": "v2-router-general-08-components", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-08-components.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-08-components.md new file mode 100644 index 0000000..73958b3 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-08-components.md @@ -0,0 +1,113 @@ +# v2-router-general-08-components + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Из чего состоит Telegram Notify App? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Из чего состоит Telegram Notify App?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Из чего состоит Telegram Notify App?", + "normalized_query": "Из чего состоит Telegram Notify App?", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "deterministic signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Из чего состоит Telegram Notify App?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "deterministic signal", + "target_terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "anchors": { + "entity_names": [ + "Telegram", + "Notify", + "App" + ], + "terms": [ + "Telegram", + "Notify", + "App", + "чего", + "состоит" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json new file mode 100644 index 0000000..cca6d29 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-09-reading-order.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v2-router-general-09-reading-order", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md new file mode 100644 index 0000000..02662d1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-09-reading-order.md @@ -0,0 +1,93 @@ +# v2-router-general-09-reading-order + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Какие документы стоит читать сначала? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Какие документы стоит читать сначала?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Какие документы стоит читать сначала?", + "normalized_query": "Какие документы стоит читать сначала?", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Какие документы стоит читать сначала?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "стоит", + "читать", + "сначала" + ], + "anchors": { + "entity_names": [], + "terms": [ + "стоит", + "читать", + "сначала" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json new file mode 100644 index 0000000..ce8e869 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-10-short-summary.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v2-router-general-10-short-summary", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 + }, + "details": { + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md new file mode 100644 index 0000000..be3dbe5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/router_only_docs_v2_matrix_v2-router-general-10-short-summary.md @@ -0,0 +1,97 @@ +# v2-router-general-10-short-summary + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml +- passed: True + +## Query +Дай короткое summary по проектной документации + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "deterministic", + "llm_router_used": false, + "confidence": 1.0 +} +``` + +## Details +```json +{ + "query": "Дай короткое summary по проектной документации", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Дай короткое summary по проектной документации", + "normalized_query": "Дай короткое summary по проектной документации", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 1.0, + "routing_mode": "deterministic", + "llm_router_used": false, + "reason_short": "general fallback signal" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Дай короткое summary по проектной документации" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general fallback signal", + "target_terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "anchors": { + "entity_names": [], + "terms": [ + "Дай", + "короткое", + "summary", + "проектной" + ], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/summary.md b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/summary.md new file mode 100644 index 0000000..966c540 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/test_runs/smoke_v4_pass/20260407_120104/summary.md @@ -0,0 +1,36 @@ +# pipeline_setup_v4 summary + +Passed: 30/30 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v2-router-general-01-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-02-purpose | process_v2_intent_router | Для чего нужен test_echo_app? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-03-problem | process_v2_intent_router | Какую задачу решает это приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-04-docs-contents | process_v2_intent_router | Что входит в документацию этого проекта? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-05-main-parts | process_v2_intent_router | Какие основные части есть у системы? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-06-high-level-architecture | process_v2_intent_router | Как в целом устроено приложение? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-07-besides-api | process_v2_intent_router | Что тут есть кроме API? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-08-components | process_v2_intent_router | Из чего состоит Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-general-09-reading-order | process_v2_intent_router | Какие документы стоит читать сначала? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-general-10-short-summary | process_v2_intent_router | Дай короткое summary по проектной документации | GENERAL_QA | SUMMARY | ✓ | +| v2-router-docs-summary-01-health | process_v2_intent_router | Кратко объясни по документации, как работает `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-02-send | process_v2_intent_router | Что делает endpoint `/send`? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-03-actions | process_v2_intent_router | Объясни назначение `/actions/{action}` | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-04-runtime-health | process_v2_intent_router | Что такое runtime health в этой документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-05-notification-loop | process_v2_intent_router | Кратко опиши цикл отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-06-architecture | process_v2_intent_router | Как устроена архитектура Telegram Notify App? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-07-worker | process_v2_intent_router | Что делает worker в этом приложении? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-08-health-statuses | process_v2_intent_router | Какие статусы здоровья есть у runtime? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-09-manual-send | process_v2_intent_router | Как в приложении устроена ручная отправка сообщения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v2-router-docs-summary-10-telegram-integration | process_v2_intent_router | Как сервис взаимодействует с Telegram в целом? | GENERAL_QA | SUMMARY | ✓ | +| v2-router-find-files-01-health | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-02-send | process_v2_intent_router | Покажи файл с описанием endpoint `/send` | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-03-actions | process_v2_intent_router | Где находится документация по `/actions/{action}`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-04-runtime-health | process_v2_intent_router | В каком документе описан runtime health? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-05-loop | process_v2_intent_router | Найди файл с логикой цикла отправки уведомлений | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-06-architecture | process_v2_intent_router | Где описана архитектура приложения? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-07-errors-catalog | process_v2_intent_router | В каком файле лежит каталог ошибок? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-08-docs-index | process_v2_intent_router | Покажи файл-индекс документации проекта | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-09-telegram-bot-api | process_v2_intent_router | Какие файлы относятся к Telegram Bot API интеграции? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v2-router-find-files-10-telegram-notify-worker | process_v2_intent_router | В каких файлах описан worker `telegram_notify`? | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml new file mode 100644 index 0000000..69f9328 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml @@ -0,0 +1,147 @@ +defaults: + component: process_v2_intent_router + +cases: + - id: v3-docs-summary-01-endpoint + query: "Объясни по документации endpoint `/health`" + tags: ["docs", "summary", "clean"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v3-docs-summary-02-architecture + query: "Как устроена архитектура приложения?" + tags: ["docs", "summary", "clean"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v3-docs-summary-03-process + query: "Опиши процесс отправки уведомлений" + tags: ["docs", "summary", "clean"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v3-docs-summary-04-entity + query: "Что такое runtime health в документации?" + tags: ["docs", "summary", "clean"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v3-find-files-01-which-file + query: "В каком файле описан `/health`?" + tags: ["docs", "find_files", "clean"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v3-find-files-02-show-doc + query: "Покажи документ про runtime health" + tags: ["docs", "find_files", "clean"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v3-find-files-03-where-described + query: "Где описано `docs/architecture/overview.md`?" + tags: ["docs", "find_files", "clean"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v3-general-01-overview + query: "Общий обзор сервиса" + tags: ["general", "summary", "clean"] + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v3-general-02-what-is-service + query: "Что это за сервис?" + tags: ["general", "summary", "clean"] + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v3-general-03-purpose + query: "Для чего нужен этот сервис?" + tags: ["general", "summary", "clean"] + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v3-negative-01-operational + query: "Почему у меня сейчас упал деплой?" + tags: ["negative", "boundary"] + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v3-negative-02-meta + query: "Как ты выбираешь маршрут?" + tags: ["negative", "boundary"] + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v3-negative-03-non-docs + query: "Сколько сейчас времени на сервере?" + tags: ["negative", "boundary"] + expected: + router: + domain: GENERAL + intent: GENERAL_QA + sub_intent: SUMMARY + + - id: v3-noisy-01-colloquial + query: "Где там дока про health, покажи плз" + tags: ["noisy"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES + + - id: v3-noisy-02-abbrev + query: "Нужен краткий док-саммари по api /send" + tags: ["noisy"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: SUMMARY + + - id: v3-noisy-03-broken-phrase + query: "runtime health это где описано в доках" + tags: ["noisy"] + expected: + router: + domain: DOCS + intent: DOC_EXPLAIN + sub_intent: FIND_FILES diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039.zip b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039.zip new file mode 100644 index 0000000..aeca4cd Binary files /dev/null and b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039.zip differ diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-01-endpoint.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-01-endpoint.json new file mode 100644 index 0000000..8ae05b6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-01-endpoint.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-docs-summary-01-endpoint", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-01-endpoint.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-01-endpoint.md new file mode 100644 index 0000000..3d3fb3c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-01-endpoint.md @@ -0,0 +1,87 @@ +# v3-docs-summary-01-endpoint + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Объясни по документации endpoint `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-02-architecture.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-02-architecture.json new file mode 100644 index 0000000..45c3a10 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-02-architecture.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-docs-summary-02-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-02-architecture.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-02-architecture.md new file mode 100644 index 0000000..808da5e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-02-architecture.md @@ -0,0 +1,91 @@ +# v3-docs-summary-02-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как устроена архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-03-process.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-03-process.json new file mode 100644 index 0000000..9b74ef1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-03-process.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-docs-summary-03-process", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-03-process.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-03-process.md new file mode 100644 index 0000000..ff2706e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-03-process.md @@ -0,0 +1,85 @@ +# v3-docs-summary-03-process + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Опиши процесс отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "опиши", + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-04-entity.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-04-entity.json new file mode 100644 index 0000000..6610cd1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-04-entity.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-docs-summary-04-entity", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-04-entity.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-04-entity.md new file mode 100644 index 0000000..9af2b71 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-docs-summary-04-entity.md @@ -0,0 +1,91 @@ +# v3-docs-summary-04-entity + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что такое runtime health в документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-01-which-file.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-01-which-file.json new file mode 100644 index 0000000..3255472 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-01-which-file.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-01-which-file", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-01-which-file.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-01-which-file.md new file mode 100644 index 0000000..6829cb7 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-01-which-file.md @@ -0,0 +1,93 @@ +# v3-find-files-01-which-file + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "каком", + "файле", + "описан" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-02-show-doc.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-02-show-doc.json new file mode 100644 index 0000000..eae070d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-02-show-doc.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-find-files-02-show-doc", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-02-show-doc.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-02-show-doc.md new file mode 100644 index 0000000..6a511a5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-02-show-doc.md @@ -0,0 +1,91 @@ +# v3-find-files-02-show-doc + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Покажи документ про runtime health + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "про", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-03-where-described.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-03-where-described.json new file mode 100644 index 0000000..4dd2e53 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-03-where-described.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-find-files-03-where-described", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-03-where-described.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-03-where-described.md new file mode 100644 index 0000000..d4d7b11 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-find-files-03-where-described.md @@ -0,0 +1,91 @@ +# v3-find-files-03-where-described + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где описано `docs/architecture/overview.md`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-01-overview.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-01-overview.json new file mode 100644 index 0000000..99d8f43 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-01-overview.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-01-overview", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-01-overview.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-01-overview.md new file mode 100644 index 0000000..a15f993 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-01-overview.md @@ -0,0 +1,83 @@ +# v3-general-01-overview + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Общий обзор сервиса + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-02-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-02-what-is-service.json new file mode 100644 index 0000000..365dcec --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-02-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-02-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-02-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-02-what-is-service.md new file mode 100644 index 0000000..a9d12ac --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-02-what-is-service.md @@ -0,0 +1,79 @@ +# v3-general-02-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-03-purpose.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-03-purpose.json new file mode 100644 index 0000000..b93b55f --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-03-purpose.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-03-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-03-purpose.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-03-purpose.md new file mode 100644 index 0000000..7454584 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-general-03-purpose.md @@ -0,0 +1,83 @@ +# v3-general-03-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Для чего нужен этот сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "чего", + "нужен", + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-01-operational.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-01-operational.json new file mode 100644 index 0000000..0ed3031 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-01-operational.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-negative-01-operational", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-01-operational.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-01-operational.md new file mode 100644 index 0000000..ef8758d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-01-operational.md @@ -0,0 +1,87 @@ +# v3-negative-01-operational + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Почему у меня сейчас упал деплой? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-02-meta.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-02-meta.json new file mode 100644 index 0000000..6650547 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-02-meta.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v3-negative-02-meta", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-02-meta.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-02-meta.md new file mode 100644 index 0000000..72dce76 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-02-meta.md @@ -0,0 +1,81 @@ +# v3-negative-02-meta + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как ты выбираешь маршрут? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-03-non-docs.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-03-non-docs.json new file mode 100644 index 0000000..6a11483 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-03-non-docs.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-negative-03-non-docs", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-03-non-docs.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-03-non-docs.md new file mode 100644 index 0000000..65fcf51 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-negative-03-non-docs.md @@ -0,0 +1,85 @@ +# v3-negative-03-non-docs + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Сколько сейчас времени на сервере? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-01-colloquial.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-01-colloquial.json new file mode 100644 index 0000000..d0f32f1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-01-colloquial.json @@ -0,0 +1,87 @@ +{ + "meta": { + "case_id": "v3-noisy-01-colloquial", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-01-colloquial.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-01-colloquial.md new file mode 100644 index 0000000..f797bdb --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-01-colloquial.md @@ -0,0 +1,97 @@ +# v3-noisy-01-colloquial + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где там дока про health, покажи плз + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health", + "там", + "дока", + "про", + "health", + "плз" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-02-abbrev.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-02-abbrev.json new file mode 100644 index 0000000..f9fa085 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-02-abbrev.json @@ -0,0 +1,85 @@ +{ + "meta": { + "case_id": "v3-noisy-02-abbrev", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-02-abbrev.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-02-abbrev.md new file mode 100644 index 0000000..f07f934 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-02-abbrev.md @@ -0,0 +1,95 @@ +# v3-noisy-02-abbrev + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Нужен краткий док-саммари по api /send + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "нужен", + "краткий", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-03-broken-phrase.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-03-broken-phrase.json new file mode 100644 index 0000000..1581952 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-03-broken-phrase.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-noisy-03-broken-phrase", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-03-broken-phrase.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-03-broken-phrase.md new file mode 100644 index 0000000..13e57b2 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/router_llm_first_v3_v3-noisy-03-broken-phrase.md @@ -0,0 +1,93 @@ +# v3-noisy-03-broken-phrase + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +runtime health это где описано в доках + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health", + "описано", + "доках" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/summary.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/summary.md new file mode 100644 index 0000000..fee2567 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_suite_check/20260407_130039/summary.md @@ -0,0 +1,22 @@ +# pipeline_setup_v4 summary + +Passed: 16/16 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v3-docs-summary-01-endpoint | process_v2_intent_router | Объясни по документации endpoint `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-02-architecture | process_v2_intent_router | Как устроена архитектура приложения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-03-process | process_v2_intent_router | Опиши процесс отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-04-entity | process_v2_intent_router | Что такое runtime health в документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-find-files-01-which-file | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-02-show-doc | process_v2_intent_router | Покажи документ про runtime health | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-03-where-described | process_v2_intent_router | Где описано `docs/architecture/overview.md`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-general-01-overview | process_v2_intent_router | Общий обзор сервиса | GENERAL_QA | SUMMARY | ✓ | +| v3-general-02-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-general-03-purpose | process_v2_intent_router | Для чего нужен этот сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-01-operational | process_v2_intent_router | Почему у меня сейчас упал деплой? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-02-meta | process_v2_intent_router | Как ты выбираешь маршрут? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-03-non-docs | process_v2_intent_router | Сколько сейчас времени на сервере? | GENERAL_QA | SUMMARY | ✓ | +| v3-noisy-01-colloquial | process_v2_intent_router | Где там дока про health, покажи плз | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-noisy-02-abbrev | process_v2_intent_router | Нужен краткий док-саммари по api /send | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-noisy-03-broken-phrase | process_v2_intent_router | runtime health это где описано в доках | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-01-endpoint.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-01-endpoint.json new file mode 100644 index 0000000..8ae05b6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-01-endpoint.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-docs-summary-01-endpoint", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-01-endpoint.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-01-endpoint.md new file mode 100644 index 0000000..3d3fb3c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-01-endpoint.md @@ -0,0 +1,87 @@ +# v3-docs-summary-01-endpoint + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Объясни по документации endpoint `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-02-architecture.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-02-architecture.json new file mode 100644 index 0000000..02e64fd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-02-architecture.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-02-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-02-architecture.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-02-architecture.md new file mode 100644 index 0000000..c684f80 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-02-architecture.md @@ -0,0 +1,93 @@ +# v3-docs-summary-02-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как устроена архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-03-process.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-03-process.json new file mode 100644 index 0000000..a0f6f41 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-03-process.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-docs-summary-03-process", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-03-process.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-03-process.md new file mode 100644 index 0000000..c99a321 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-03-process.md @@ -0,0 +1,83 @@ +# v3-docs-summary-03-process + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Опиши процесс отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-04-entity.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-04-entity.json new file mode 100644 index 0000000..6610cd1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-04-entity.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-docs-summary-04-entity", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-04-entity.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-04-entity.md new file mode 100644 index 0000000..9af2b71 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-docs-summary-04-entity.md @@ -0,0 +1,91 @@ +# v3-docs-summary-04-entity + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что такое runtime health в документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-01-which-file.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-01-which-file.json new file mode 100644 index 0000000..56f9c1a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-01-which-file.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-find-files-01-which-file", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-01-which-file.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-01-which-file.md new file mode 100644 index 0000000..06dd103 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-01-which-file.md @@ -0,0 +1,87 @@ +# v3-find-files-01-which-file + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-02-show-doc.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-02-show-doc.json new file mode 100644 index 0000000..3bb3db5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-02-show-doc.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v3-find-files-02-show-doc", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-02-show-doc.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-02-show-doc.md new file mode 100644 index 0000000..0768c83 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-02-show-doc.md @@ -0,0 +1,89 @@ +# v3-find-files-02-show-doc + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Покажи документ про runtime health + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-03-where-described.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-03-where-described.json new file mode 100644 index 0000000..e5db934 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-03-where-described.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-03-where-described", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-03-where-described.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-03-where-described.md new file mode 100644 index 0000000..4dfb90c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-find-files-03-where-described.md @@ -0,0 +1,93 @@ +# v3-find-files-03-where-described + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где описано `docs/architecture/overview.md`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-01-overview.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-01-overview.json new file mode 100644 index 0000000..99d8f43 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-01-overview.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-01-overview", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-01-overview.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-01-overview.md new file mode 100644 index 0000000..a15f993 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-01-overview.md @@ -0,0 +1,83 @@ +# v3-general-01-overview + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Общий обзор сервиса + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-02-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-02-what-is-service.json new file mode 100644 index 0000000..365dcec --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-02-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-02-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-02-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-02-what-is-service.md new file mode 100644 index 0000000..a9d12ac --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-02-what-is-service.md @@ -0,0 +1,79 @@ +# v3-general-02-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-03-purpose.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-03-purpose.json new file mode 100644 index 0000000..26300a0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-03-purpose.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-03-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-03-purpose.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-03-purpose.md new file mode 100644 index 0000000..11186f9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-general-03-purpose.md @@ -0,0 +1,79 @@ +# v3-general-03-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Для чего нужен этот сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-01-operational.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-01-operational.json new file mode 100644 index 0000000..0ed3031 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-01-operational.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-negative-01-operational", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-01-operational.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-01-operational.md new file mode 100644 index 0000000..ef8758d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-01-operational.md @@ -0,0 +1,87 @@ +# v3-negative-01-operational + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Почему у меня сейчас упал деплой? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-02-meta.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-02-meta.json new file mode 100644 index 0000000..6650547 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-02-meta.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v3-negative-02-meta", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-02-meta.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-02-meta.md new file mode 100644 index 0000000..72dce76 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-02-meta.md @@ -0,0 +1,81 @@ +# v3-negative-02-meta + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как ты выбираешь маршрут? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-03-non-docs.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-03-non-docs.json new file mode 100644 index 0000000..6a11483 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-03-non-docs.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-negative-03-non-docs", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-03-non-docs.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-03-non-docs.md new file mode 100644 index 0000000..65fcf51 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-negative-03-non-docs.md @@ -0,0 +1,85 @@ +# v3-negative-03-non-docs + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Сколько сейчас времени на сервере? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-01-colloquial.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-01-colloquial.json new file mode 100644 index 0000000..3458b13 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-01-colloquial.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-noisy-01-colloquial", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-01-colloquial.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-01-colloquial.md new file mode 100644 index 0000000..acb5111 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-01-colloquial.md @@ -0,0 +1,79 @@ +# v3-noisy-01-colloquial + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где там дока про health, покажи плз + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-02-abbrev.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-02-abbrev.json new file mode 100644 index 0000000..4af000e --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-02-abbrev.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-noisy-02-abbrev", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-02-abbrev.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-02-abbrev.md new file mode 100644 index 0000000..4df06da --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-02-abbrev.md @@ -0,0 +1,91 @@ +# v3-noisy-02-abbrev + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Нужен краткий док-саммари по api /send + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "док-саммари", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-03-broken-phrase.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-03-broken-phrase.json new file mode 100644 index 0000000..af51d3d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-03-broken-phrase.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-noisy-03-broken-phrase", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-03-broken-phrase.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-03-broken-phrase.md new file mode 100644 index 0000000..d69e056 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/router_llm_first_v3_v3-noisy-03-broken-phrase.md @@ -0,0 +1,91 @@ +# v3-noisy-03-broken-phrase + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +runtime health это где описано в доках + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/summary.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/summary.md new file mode 100644 index 0000000..fee2567 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_check/20260407_140409/summary.md @@ -0,0 +1,22 @@ +# pipeline_setup_v4 summary + +Passed: 16/16 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v3-docs-summary-01-endpoint | process_v2_intent_router | Объясни по документации endpoint `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-02-architecture | process_v2_intent_router | Как устроена архитектура приложения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-03-process | process_v2_intent_router | Опиши процесс отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-04-entity | process_v2_intent_router | Что такое runtime health в документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-find-files-01-which-file | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-02-show-doc | process_v2_intent_router | Покажи документ про runtime health | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-03-where-described | process_v2_intent_router | Где описано `docs/architecture/overview.md`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-general-01-overview | process_v2_intent_router | Общий обзор сервиса | GENERAL_QA | SUMMARY | ✓ | +| v3-general-02-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-general-03-purpose | process_v2_intent_router | Для чего нужен этот сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-01-operational | process_v2_intent_router | Почему у меня сейчас упал деплой? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-02-meta | process_v2_intent_router | Как ты выбираешь маршрут? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-03-non-docs | process_v2_intent_router | Сколько сейчас времени на сервере? | GENERAL_QA | SUMMARY | ✓ | +| v3-noisy-01-colloquial | process_v2_intent_router | Где там дока про health, покажи плз | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-noisy-02-abbrev | process_v2_intent_router | Нужен краткий док-саммари по api /send | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-noisy-03-broken-phrase | process_v2_intent_router | runtime health это где описано в доках | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-01-endpoint.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-01-endpoint.json new file mode 100644 index 0000000..8ae05b6 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-01-endpoint.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-docs-summary-01-endpoint", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-01-endpoint.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-01-endpoint.md new file mode 100644 index 0000000..3d3fb3c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-01-endpoint.md @@ -0,0 +1,87 @@ +# v3-docs-summary-01-endpoint + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Объясни по документации endpoint `/health` + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Объясни по документации endpoint `/health`", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Объясни по документации endpoint `/health`", + "normalized_query": "Объясни по документации endpoint `/health`", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Объясни по документации endpoint `/health`" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-02-architecture.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-02-architecture.json new file mode 100644 index 0000000..02e64fd --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-02-architecture.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-docs-summary-02-architecture", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-02-architecture.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-02-architecture.md new file mode 100644 index 0000000..c684f80 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-02-architecture.md @@ -0,0 +1,93 @@ +# v3-docs-summary-02-architecture + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как устроена архитектура приложения? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Как устроена архитектура приложения?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Как устроена архитектура приложения?", + "normalized_query": "Как устроена архитектура приложения?", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как устроена архитектура приложения?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "architecture_overview", + "устроена", + "архитектура", + "приложения" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-03-process.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-03-process.json new file mode 100644 index 0000000..a0f6f41 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-03-process.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-docs-summary-03-process", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-03-process.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-03-process.md new file mode 100644 index 0000000..c99a321 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-03-process.md @@ -0,0 +1,83 @@ +# v3-docs-summary-03-process + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Опиши процесс отправки уведомлений + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Опиши процесс отправки уведомлений", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Опиши процесс отправки уведомлений", + "normalized_query": "Опиши процесс отправки уведомлений", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Опиши процесс отправки уведомлений" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "процесс", + "отправки", + "уведомлений" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-04-entity.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-04-entity.json new file mode 100644 index 0000000..6610cd1 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-04-entity.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-docs-summary-04-entity", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 + }, + "details": { + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-04-entity.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-04-entity.md new file mode 100644 index 0000000..9af2b71 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-docs-summary-04-entity.md @@ -0,0 +1,91 @@ +# v3-docs-summary-04-entity + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что такое runtime health в документации? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.93 +} +``` + +## Details +```json +{ + "query": "Что такое runtime health в документации?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Что такое runtime health в документации?", + "normalized_query": "Что такое runtime health в документации?", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.93, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что такое runtime health в документации?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "такое", + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-01-which-file.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-01-which-file.json new file mode 100644 index 0000000..56f9c1a --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-01-which-file.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-find-files-01-which-file", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-01-which-file.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-01-which-file.md new file mode 100644 index 0000000..06dd103 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-01-which-file.md @@ -0,0 +1,87 @@ +# v3-find-files-01-which-file + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +В каком файле описан `/health`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "В каком файле описан `/health`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "В каком файле описан `/health`?", + "normalized_query": "В каком файле описан `/health`?", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "В каком файле описан `/health`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "/health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/health" + ], + "target_doc_hints": [ + "docs/api/health-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-02-show-doc.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-02-show-doc.json new file mode 100644 index 0000000..3bb3db5 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-02-show-doc.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v3-find-files-02-show-doc", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-02-show-doc.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-02-show-doc.md new file mode 100644 index 0000000..0768c83 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-02-show-doc.md @@ -0,0 +1,89 @@ +# v3-find-files-02-show-doc + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Покажи документ про runtime health + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Покажи документ про runtime health", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Покажи документ про runtime health", + "normalized_query": "Покажи документ про runtime health", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Покажи документ про runtime health" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-03-where-described.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-03-where-described.json new file mode 100644 index 0000000..e5db934 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-03-where-described.json @@ -0,0 +1,83 @@ +{ + "meta": { + "case_id": "v3-find-files-03-where-described", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 + }, + "details": { + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-03-where-described.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-03-where-described.md new file mode 100644 index 0000000..4dfb90c --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-find-files-03-where-described.md @@ -0,0 +1,93 @@ +# v3-find-files-03-where-described + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где описано `docs/architecture/overview.md`? + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8800000000000001 +} +``` + +## Details +```json +{ + "query": "Где описано `docs/architecture/overview.md`?", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где описано `docs/architecture/overview.md`?", + "normalized_query": "Где описано `docs/architecture/overview.md`?", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где описано `docs/architecture/overview.md`?" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "architecture_overview", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [ + "docs/architecture/overview.md" + ], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/architecture/telegram-notify-app-overview.md" + ], + "matched_aliases": [ + "architecture_overview" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-01-overview.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-01-overview.json new file mode 100644 index 0000000..99d8f43 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-01-overview.json @@ -0,0 +1,73 @@ +{ + "meta": { + "case_id": "v3-general-01-overview", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-01-overview.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-01-overview.md new file mode 100644 index 0000000..a15f993 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-01-overview.md @@ -0,0 +1,83 @@ +# v3-general-01-overview + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Общий обзор сервиса + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Общий обзор сервиса", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Общий обзор сервиса", + "normalized_query": "Общий обзор сервиса", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Общий обзор сервиса" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "общий", + "обзор", + "сервиса" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-02-what-is-service.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-02-what-is-service.json new file mode 100644 index 0000000..365dcec --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-02-what-is-service.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-02-what-is-service", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-02-what-is-service.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-02-what-is-service.md new file mode 100644 index 0000000..a9d12ac --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-02-what-is-service.md @@ -0,0 +1,79 @@ +# v3-general-02-what-is-service + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Что это за сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Что это за сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Что это за сервис?", + "normalized_query": "Что это за сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Что это за сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-03-purpose.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-03-purpose.json new file mode 100644 index 0000000..26300a0 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-03-purpose.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-general-03-purpose", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-03-purpose.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-03-purpose.md new file mode 100644 index 0000000..11186f9 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-general-03-purpose.md @@ -0,0 +1,79 @@ +# v3-general-03-purpose + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Для чего нужен этот сервис? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Для чего нужен этот сервис?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Для чего нужен этот сервис?", + "normalized_query": "Для чего нужен этот сервис?", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "general overview" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Для чего нужен этот сервис?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "general overview", + "target_terms": [ + "сервис" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-01-operational.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-01-operational.json new file mode 100644 index 0000000..0ed3031 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-01-operational.json @@ -0,0 +1,77 @@ +{ + "meta": { + "case_id": "v3-negative-01-operational", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-01-operational.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-01-operational.md new file mode 100644 index 0000000..ef8758d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-01-operational.md @@ -0,0 +1,87 @@ +# v3-negative-01-operational + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Почему у меня сейчас упал деплой? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Почему у меня сейчас упал деплой?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Почему у меня сейчас упал деплой?", + "normalized_query": "Почему у меня сейчас упал деплой?", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Почему у меня сейчас упал деплой?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "почему", + "меня", + "сейчас", + "упал", + "деплой" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-02-meta.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-02-meta.json new file mode 100644 index 0000000..6650547 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-02-meta.json @@ -0,0 +1,71 @@ +{ + "meta": { + "case_id": "v3-negative-02-meta", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-02-meta.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-02-meta.md new file mode 100644 index 0000000..72dce76 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-02-meta.md @@ -0,0 +1,81 @@ +# v3-negative-02-meta + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Как ты выбираешь маршрут? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Как ты выбираешь маршрут?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Как ты выбираешь маршрут?", + "normalized_query": "Как ты выбираешь маршрут?", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Как ты выбираешь маршрут?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "выбираешь", + "маршрут" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-03-non-docs.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-03-non-docs.json new file mode 100644 index 0000000..6a11483 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-03-non-docs.json @@ -0,0 +1,75 @@ +{ + "meta": { + "case_id": "v3-negative-03-non-docs", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 + }, + "details": { + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-03-non-docs.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-03-non-docs.md new file mode 100644 index 0000000..65fcf51 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-negative-03-non-docs.md @@ -0,0 +1,85 @@ +# v3-negative-03-non-docs + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Сколько сейчас времени на сервере? + +## Actual +```json +{ + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.8300000000000001 +} +``` + +## Details +```json +{ + "query": "Сколько сейчас времени на сервере?", + "route": { + "routing_domain": "GENERAL", + "intent": "GENERAL_QA", + "subintent": "SUMMARY", + "user_query": "Сколько сейчас времени на сервере?", + "normalized_query": "Сколько сейчас времени на сервере?", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.8300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "default general" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Сколько сейчас времени на сервере?" + }, + "output": { + "domain": "GENERAL", + "intent": "GENERAL_QA", + "sub_intent": "SUMMARY", + "reason_short": "default general", + "target_terms": [ + "сколько", + "сейчас", + "времени", + "сервере" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-01-colloquial.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-01-colloquial.json new file mode 100644 index 0000000..3458b13 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-01-colloquial.json @@ -0,0 +1,69 @@ +{ + "meta": { + "case_id": "v3-noisy-01-colloquial", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 + }, + "details": { + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-01-colloquial.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-01-colloquial.md new file mode 100644 index 0000000..acb5111 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-01-colloquial.md @@ -0,0 +1,79 @@ +# v3-noisy-01-colloquial + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Где там дока про health, покажи плз + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.7300000000000001 +} +``` + +## Details +```json +{ + "query": "Где там дока про health, покажи плз", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "Где там дока про health, покажи плз", + "normalized_query": "Где там дока про health, покажи плз", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.7300000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Где там дока про health, покажи плз" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "health" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-02-abbrev.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-02-abbrev.json new file mode 100644 index 0000000..d7d7b67 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-02-abbrev.json @@ -0,0 +1,79 @@ +{ + "meta": { + "case_id": "v3-noisy-02-abbrev", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-02-abbrev.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-02-abbrev.md new file mode 100644 index 0000000..5962391 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-02-abbrev.md @@ -0,0 +1,89 @@ +# v3-noisy-02-abbrev + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +Нужен краткий док-саммари по api /send + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "Нужен краткий док-саммари по api /send", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "SUMMARY", + "user_query": "Нужен краткий док-саммари по api /send", + "normalized_query": "Нужен краткий док-саммари по api /send", + "target_terms": [ + "/send", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "docs topic" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "Нужен краткий док-саммари по api /send" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "SUMMARY", + "reason_short": "docs topic", + "target_terms": [ + "/send", + "api" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [ + "/send" + ], + "target_doc_hints": [ + "docs/api/send-message-endpoint.md" + ], + "matched_aliases": [], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-03-broken-phrase.json b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-03-broken-phrase.json new file mode 100644 index 0000000..af51d3d --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-03-broken-phrase.json @@ -0,0 +1,81 @@ +{ + "meta": { + "case_id": "v3-noisy-03-broken-phrase", + "component": "process_v2_intent_router", + "source_file": "/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml", + "passed": true, + "mismatches": [] + }, + "actual": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 + }, + "details": { + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] + } +} \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-03-broken-phrase.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-03-broken-phrase.md new file mode 100644 index 0000000..d69e056 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/router_llm_first_v3_v3-noisy-03-broken-phrase.md @@ -0,0 +1,91 @@ +# v3-noisy-03-broken-phrase + +- component: process_v2_intent_router +- source_file: /Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml +- passed: True + +## Query +runtime health это где описано в доках + +## Actual +```json +{ + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "routing_mode": "llm_default", + "llm_router_used": true, + "confidence": 0.9800000000000001 +} +``` + +## Details +```json +{ + "query": "runtime health это где описано в доках", + "route": { + "routing_domain": "DOCS", + "intent": "DOC_EXPLAIN", + "subintent": "FIND_FILES", + "user_query": "runtime health это где описано в доках", + "normalized_query": "runtime health это где описано в доках", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + }, + "confidence": 0.9800000000000001, + "routing_mode": "llm_default", + "llm_router_used": true, + "reason_short": "file lookup" + }, + "pipeline_steps": [ + { + "step": "intent_router", + "input": { + "query": "runtime health это где описано в доках" + }, + "output": { + "domain": "DOCS", + "intent": "DOC_EXPLAIN", + "sub_intent": "FIND_FILES", + "reason_short": "file lookup", + "target_terms": [ + "runtime", + "health", + "описано" + ], + "anchors": { + "entity_names": [], + "file_names": [], + "endpoint_paths": [], + "target_doc_hints": [ + "docs/domains/runtime-health-entity.md" + ], + "matched_aliases": [ + "runtime_health" + ], + "process_domain": null, + "process_subdomain": null + } + } + } + ] +} +``` + +## Mismatches +- none \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/summary.md b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/summary.md new file mode 100644 index 0000000..fee2567 --- /dev/null +++ b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/llm_first_v3_target_terms_final/20260407_140431/summary.md @@ -0,0 +1,22 @@ +# pipeline_setup_v4 summary + +Passed: 16/16 + +| Case | Component | Query | Intent | Sub-intent | Pass | +|------|-----------|-------|--------|------------|------| +| v3-docs-summary-01-endpoint | process_v2_intent_router | Объясни по документации endpoint `/health` | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-02-architecture | process_v2_intent_router | Как устроена архитектура приложения? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-03-process | process_v2_intent_router | Опиши процесс отправки уведомлений | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-docs-summary-04-entity | process_v2_intent_router | Что такое runtime health в документации? | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-find-files-01-which-file | process_v2_intent_router | В каком файле описан `/health`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-02-show-doc | process_v2_intent_router | Покажи документ про runtime health | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-find-files-03-where-described | process_v2_intent_router | Где описано `docs/architecture/overview.md`? | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-general-01-overview | process_v2_intent_router | Общий обзор сервиса | GENERAL_QA | SUMMARY | ✓ | +| v3-general-02-what-is-service | process_v2_intent_router | Что это за сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-general-03-purpose | process_v2_intent_router | Для чего нужен этот сервис? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-01-operational | process_v2_intent_router | Почему у меня сейчас упал деплой? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-02-meta | process_v2_intent_router | Как ты выбираешь маршрут? | GENERAL_QA | SUMMARY | ✓ | +| v3-negative-03-non-docs | process_v2_intent_router | Сколько сейчас времени на сервере? | GENERAL_QA | SUMMARY | ✓ | +| v3-noisy-01-colloquial | process_v2_intent_router | Где там дока про health, покажи плз | DOC_EXPLAIN | FIND_FILES | ✓ | +| v3-noisy-02-abbrev | process_v2_intent_router | Нужен краткий док-саммари по api /send | DOC_EXPLAIN | SUMMARY | ✓ | +| v3-noisy-03-broken-phrase | process_v2_intent_router | runtime health это где описано в доках | DOC_EXPLAIN | FIND_FILES | ✓ | \ No newline at end of file diff --git a/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/Архив.zip b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/Архив.zip new file mode 100644 index 0000000..a8222db Binary files /dev/null and b/tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/test_runs/Архив.zip differ diff --git a/tests/pipeline_setup_v4/core/__init__.py b/tests/pipeline_setup_v4/core/__init__.py new file mode 100644 index 0000000..96c67fa --- /dev/null +++ b/tests/pipeline_setup_v4/core/__init__.py @@ -0,0 +1 @@ +"""Core helpers for pipeline_setup_v4.""" diff --git a/tests/pipeline_setup_v4/core/artifacts.py b/tests/pipeline_setup_v4/core/artifacts.py new file mode 100644 index 0000000..a54410f --- /dev/null +++ b/tests/pipeline_setup_v4/core/artifacts.py @@ -0,0 +1,114 @@ +from __future__ import annotations + +import json +from collections import defaultdict +from datetime import datetime +from pathlib import Path + +from tests.pipeline_setup_v4.core.models import V4CaseResult + + +class ArtifactLayout: + def __init__(self, *, run_name: str, started_at: datetime) -> None: + self._run_name = run_name + self._stamp = started_at.strftime("%Y%m%d_%H%M%S") + + def run_dir_for(self, source_file: Path) -> Path: + path = source_file.parent / "test_runs" / self._run_name / self._stamp + path.mkdir(parents=True, exist_ok=True) + return path + + +class ArtifactWriter: + def __init__(self, layout: ArtifactLayout) -> None: + self._layout = layout + + def write_case(self, result: V4CaseResult) -> Path: + run_dir = self._layout.run_dir_for(result.case.source_file) + stem = f"{result.case.source_file.stem}_{result.case.case_id}" + json_path = run_dir / f"{stem}.json" + md_path = run_dir / f"{stem}.md" + json_path.write_text(json.dumps(self._json_payload(result), ensure_ascii=False, indent=2), encoding="utf-8") + md_path.write_text(self._markdown_payload(result), encoding="utf-8") + return md_path + + def write_summaries(self, results: list[V4CaseResult]) -> list[Path]: + grouped: dict[Path, list[V4CaseResult]] = defaultdict(list) + for result in results: + grouped[self._layout.run_dir_for(result.case.source_file)].append(result) + paths: list[Path] = [] + for run_dir, items in sorted(grouped.items(), key=lambda item: item[0].as_posix()): + path = run_dir / "summary.md" + path.write_text(SummaryComposer().compose(items), encoding="utf-8") + paths.append(path) + return paths + + def _json_payload(self, result: V4CaseResult) -> dict: + return { + "meta": { + "case_id": result.case.case_id, + "component": result.case.component, + "source_file": result.case.source_file.as_posix(), + "passed": result.passed, + "mismatches": result.mismatches, + }, + "actual": result.actual, + "details": result.details, + } + + def _markdown_payload(self, result: V4CaseResult) -> str: + lines = [ + f"# {result.case.case_id}", + "", + f"- component: {result.case.component}", + f"- source_file: {result.case.source_file.as_posix()}", + f"- passed: {result.passed}", + "", + "## Query", + result.case.query, + "", + "## Actual", + "```json", + json.dumps(result.actual, ensure_ascii=False, indent=2), + "```", + "", + "## Details", + "```json", + json.dumps(result.details, ensure_ascii=False, indent=2), + "```", + "", + "## Mismatches", + *([f"- {item}" for item in result.mismatches] or ["- none"]), + ] + return "\n".join(lines) + + +class SummaryComposer: + def compose(self, results: list[V4CaseResult]) -> str: + passed = sum(1 for item in results if item.passed) + lines = [ + "# pipeline_setup_v4 summary", + "", + f"Passed: {passed}/{len(results)}", + "", + "| Case | Component | Query | Intent | Sub-intent | Pass |", + "|------|-----------|-------|--------|------------|------|", + ] + for item in results: + lines.append( + f"| {item.case.case_id} | {item.case.component} | {self._cell(item.case.query)} | " + f"{item.actual.get('intent') or '—'} | {item.actual.get('sub_intent') or '—'} | " + f"{'✓' if item.passed else '✗'} |" + ) + failures = [item for item in results if not item.passed] + if failures: + lines.extend(["", "## Failures"]) + for item in failures: + lines.append(f"- **{item.case.case_id}**: {'; '.join(item.mismatches)}") + return "\n".join(lines) + + def _cell(self, text: str, limit: int = 140) -> str: + compact = " ".join(text.split()).replace("|", "\\|") + if len(compact) <= limit: + return compact + return compact[: limit - 1].rstrip() + "…" diff --git a/tests/pipeline_setup_v4/core/case_loader.py b/tests/pipeline_setup_v4/core/case_loader.py new file mode 100644 index 0000000..ba7a2ae --- /dev/null +++ b/tests/pipeline_setup_v4/core/case_loader.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +from pathlib import Path + +import yaml + +from tests.pipeline_setup_v4.core.models import CaseExpectations, RouterExpectation, V4Case + + +class CaseDirectoryLoader: + def load(self, cases_dir: Path) -> list[V4Case]: + if cases_dir.is_file(): + return self._load_file(cases_dir) + files = sorted(path for path in cases_dir.rglob("*.yaml") if path.is_file()) + if not files: + raise ValueError(f"No YAML case files found in: {cases_dir}") + cases: list[V4Case] = [] + for path in files: + cases.extend(self._load_file(path)) + return cases + + def _load_file(self, path: Path) -> list[V4Case]: + payload = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + if not isinstance(payload, dict): + raise ValueError(f"Invalid case file {path}: expected mapping") + defaults = dict(payload.get("defaults") or {}) + items = payload.get("cases") or [] + if not isinstance(items, list): + raise ValueError(f"Invalid case file {path}: `cases` must be a list") + return [self._to_case(path, raw, defaults) for raw in items] + + def _to_case(self, path: Path, raw: object, defaults: dict) -> V4Case: + if not isinstance(raw, dict): + raise ValueError(f"Invalid case in {path}: expected object") + case_id = str(raw.get("id") or "").strip() + component = str(raw.get("component") or defaults.get("component") or "").strip() + query = str(raw.get("query") or "").strip() + if not case_id or not component or not query: + raise ValueError(f"Invalid case in {path}: `id`, `component`, `query` are required") + expected = dict(raw.get("expected") or {}) + return V4Case( + case_id=case_id, + component=component, # type: ignore[arg-type] + query=query, + source_file=path, + expectations=self._to_expectations(expected), + notes=str(raw.get("notes") or ""), + tags=tuple(str(item).strip() for item in raw.get("tags") or [] if str(item).strip()), + ) + + def _to_expectations(self, raw: dict) -> CaseExpectations: + router = dict(raw.get("router") or {}) + return CaseExpectations( + router=RouterExpectation( + domain=str(router.get("domain") or "").strip() or None, + intent=str(router.get("intent") or "").strip() or None, + sub_intent=str(router.get("sub_intent") or "").strip() or None, + ) + ) diff --git a/tests/pipeline_setup_v4/core/models.py b/tests/pipeline_setup_v4/core/models.py new file mode 100644 index 0000000..2a8c27c --- /dev/null +++ b/tests/pipeline_setup_v4/core/models.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Literal + + +ComponentKind = Literal["process_v2_intent_router"] + + +@dataclass(slots=True, frozen=True) +class RouterExpectation: + domain: str | None = None + intent: str | None = None + sub_intent: str | None = None + + +@dataclass(slots=True, frozen=True) +class CaseExpectations: + router: RouterExpectation = RouterExpectation() + + +@dataclass(slots=True, frozen=True) +class V4Case: + case_id: str + component: ComponentKind + query: str + source_file: Path + expectations: CaseExpectations = CaseExpectations() + notes: str = "" + tags: tuple[str, ...] = () + + +@dataclass(slots=True, frozen=True) +class ExecutionPayload: + actual: dict + details: dict + + +@dataclass(slots=True) +class V4CaseResult: + case: V4Case + actual: dict + details: dict + passed: bool + mismatches: list[str] = field(default_factory=list) diff --git a/tests/pipeline_setup_v4/core/runner.py b/tests/pipeline_setup_v4/core/runner.py new file mode 100644 index 0000000..b87b9a5 --- /dev/null +++ b/tests/pipeline_setup_v4/core/runner.py @@ -0,0 +1,34 @@ +from __future__ import annotations + +from datetime import datetime +from pathlib import Path + +from tests.pipeline_setup_v4.core.artifacts import ArtifactLayout, ArtifactWriter +from tests.pipeline_setup_v4.core.case_loader import CaseDirectoryLoader +from tests.pipeline_setup_v4.core.models import V4CaseResult +from tests.pipeline_setup_v4.core.validators import CaseValidator +from tests.pipeline_setup_v4.executors.registry import ExecutorRegistry + + +class V4Runner: + def __init__(self, cases_dir: Path, *, run_name: str) -> None: + self._cases_dir = cases_dir + self._validator = CaseValidator() + self._executors = ExecutorRegistry() + self._writer = ArtifactWriter(ArtifactLayout(run_name=run_name, started_at=datetime.now())) + + def run(self) -> tuple[list[V4CaseResult], list[Path]]: + results: list[V4CaseResult] = [] + for case in CaseDirectoryLoader().load(self._cases_dir): + payload = self._executors.execute(case.component, case) + mismatches = self._validator.validate(case, payload.actual) + result = V4CaseResult( + case=case, + actual=payload.actual, + details=payload.details, + passed=not mismatches, + mismatches=mismatches, + ) + self._writer.write_case(result) + results.append(result) + return results, self._writer.write_summaries(results) diff --git a/tests/pipeline_setup_v4/core/validators.py b/tests/pipeline_setup_v4/core/validators.py new file mode 100644 index 0000000..e17f269 --- /dev/null +++ b/tests/pipeline_setup_v4/core/validators.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from tests.pipeline_setup_v4.core.models import V4Case + + +class CaseValidator: + def validate(self, case: V4Case, actual: dict) -> list[str]: + mismatches: list[str] = [] + expected = case.expectations.router + self._check(expected.domain, actual.get("domain"), "domain", mismatches) + self._check(expected.intent, actual.get("intent"), "intent", mismatches) + self._check(expected.sub_intent, actual.get("sub_intent"), "sub_intent", mismatches) + return mismatches + + def _check(self, expected: str | None, actual: object, label: str, mismatches: list[str]) -> None: + if expected is not None and expected != actual: + mismatches.append(f"{label}: expected {expected}, got {actual}") diff --git a/tests/pipeline_setup_v4/executors/__init__.py b/tests/pipeline_setup_v4/executors/__init__.py new file mode 100644 index 0000000..9486c89 --- /dev/null +++ b/tests/pipeline_setup_v4/executors/__init__.py @@ -0,0 +1 @@ +"""Component executors for pipeline_setup_v4.""" diff --git a/tests/pipeline_setup_v4/executors/process_v2_router_executor.py b/tests/pipeline_setup_v4/executors/process_v2_router_executor.py new file mode 100644 index 0000000..74e7d2d --- /dev/null +++ b/tests/pipeline_setup_v4/executors/process_v2_router_executor.py @@ -0,0 +1,121 @@ +from __future__ import annotations + +import json +from dataclasses import asdict + +from app.core.agent.processes.v2 import V2IntentRouter +from tests.pipeline_setup_v4.core.models import ExecutionPayload, V4Case + + +class _KeywordLlm: + _FILE_MARKERS = ( + "в каком файле", + "в каком документе", + "в каких файлах", + "какие файлы относятся", + "покажи файл", + "покажи документ", + "где описано", + "где описан", + "где описана", + "где описаны", + "где находится", + "найди файл", + "найди файлы", + ) + _DOC_MARKERS = ( + "документац", + "endpoint", + "эндпоинт", + "архитект", + "процесс", + "сущност", + "worker", + "цикл отправки уведомлений", + "ручная отправка сообщения", + "runtime health", + "здоров", + "runtime", + "health", + "telegram", + "api", + "docs/", + "/health", + "/send", + "/actions/{action}", + ) + _GENERAL_MARKERS = ( + "что это за сервис", + "для чего нужен", + "какую задачу решает", + "общий обзор", + "что входит в документацию", + "какие документы стоит читать сначала", + "дай короткое summary", + "что тут есть кроме api", + "как в целом устроено приложение", + "какие основные части есть", + "как сервис взаимодействует с telegram в целом", + "что это", + "для чего", + ) + + def generate(self, _prompt_name: str, user_input: str, **_kwargs) -> str: + payload = json.loads(user_input) + query = str(payload.get("normalized_query") or "").lower() + route = self._select(query) + return json.dumps(route, ensure_ascii=False) + + def _select(self, query: str) -> dict[str, object]: + if any(marker in query for marker in self._FILE_MARKERS) or ("дока" in query and "покажи" in query): + return self._route("DOCS", "DOC_EXPLAIN", "FIND_FILES", "file lookup") + if any(marker in query for marker in self._GENERAL_MARKERS): + return self._route("GENERAL", "GENERAL_QA", "SUMMARY", "general overview") + if any(marker in query for marker in self._DOC_MARKERS): + return self._route("DOCS", "DOC_EXPLAIN", "SUMMARY", "docs topic") + return self._route("GENERAL", "GENERAL_QA", "SUMMARY", "default general") + + def _route(self, domain: str, intent: str, subintent: str, reason: str) -> dict[str, object]: + return { + "routing_domain": domain, + "intent": intent, + "subintent": subintent, + "confidence": 0.93, + "reason_short": reason, + } + + +class ProcessV2IntentRouterExecutor: + def __init__(self) -> None: + self._router = V2IntentRouter(llm=_KeywordLlm(), enable_llm_disambiguation=True) + + def execute(self, case: V4Case) -> ExecutionPayload: + route = self._router.route(case.query) + route_dump = asdict(route) + actual = { + "domain": route.routing_domain, + "intent": route.intent, + "sub_intent": route.subintent, + "routing_mode": route.routing_mode, + "llm_router_used": route.llm_router_used, + "confidence": route.confidence, + } + details = { + "query": case.query, + "route": route_dump, + "pipeline_steps": [ + { + "step": "intent_router", + "input": {"query": case.query}, + "output": { + "domain": route.routing_domain, + "intent": route.intent, + "sub_intent": route.subintent, + "reason_short": route.reason_short, + "target_terms": list(route.target_terms), + "anchors": route_dump.get("anchors") or {}, + }, + } + ], + } + return ExecutionPayload(actual=actual, details=details) diff --git a/tests/pipeline_setup_v4/executors/registry.py b/tests/pipeline_setup_v4/executors/registry.py new file mode 100644 index 0000000..76a298b --- /dev/null +++ b/tests/pipeline_setup_v4/executors/registry.py @@ -0,0 +1,18 @@ +from __future__ import annotations + +from tests.pipeline_setup_v4.executors.process_v2_router_executor import ProcessV2IntentRouterExecutor + + +class ExecutorRegistry: + def __init__(self) -> None: + self._router_executor: ProcessV2IntentRouterExecutor | None = None + + def execute(self, component: str, case) -> object: + if component == "process_v2_intent_router": + return self._router().execute(case) + raise ValueError(f"Unsupported component: {component}") + + def _router(self) -> ProcessV2IntentRouterExecutor: + if self._router_executor is None: + self._router_executor = ProcessV2IntentRouterExecutor() + return self._router_executor diff --git a/tests/pipeline_setup_v4/run.py b/tests/pipeline_setup_v4/run.py new file mode 100644 index 0000000..e52de5f --- /dev/null +++ b/tests/pipeline_setup_v4/run.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +_agent_root = Path(__file__).resolve().parents[2] +if str(_agent_root) not in sys.path: + sys.path.insert(0, str(_agent_root)) +_src = _agent_root / "src" +if _src.exists() and str(_src) not in sys.path: + sys.path.insert(0, str(_src)) + +from tests.pipeline_setup_v4.core.runner import V4Runner +from tests.pipeline_setup_v4.shared.env_loader import load_pipeline_setup_env + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Run isolated component cases for pipeline_setup_v4") + parser.add_argument("--cases-dir", required=True, help="Directory or file with YAML case files") + parser.add_argument("--run-name", default="manual_run", help="Run directory name inside cases/*/test_runs") + ns = parser.parse_args(argv) + + cases_dir = Path(str(ns.cases_dir)).expanduser().resolve() + load_pipeline_setup_env(start_dir=Path(__file__).resolve().parent) + runner = V4Runner(cases_dir=cases_dir, run_name=str(ns.run_name).strip() or "manual_run") + print(f"Cases dir: {cases_dir}") + results, summary_paths = runner.run() + passed = sum(1 for item in results if item.passed) + print(f"Passed: {passed}/{len(results)}") + for path in summary_paths: + print(f"Summary: {path}") + for item in results: + if not item.passed: + print(f"FAIL {item.case.case_id}: {'; '.join(item.mismatches)}") + return 0 if passed == len(results) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/pipeline_setup_v4/shared/__init__.py b/tests/pipeline_setup_v4/shared/__init__.py new file mode 100644 index 0000000..67e4ac4 --- /dev/null +++ b/tests/pipeline_setup_v4/shared/__init__.py @@ -0,0 +1 @@ +"""Shared helpers for pipeline_setup_v4.""" diff --git a/tests/pipeline_setup_v4/shared/env_loader.py b/tests/pipeline_setup_v4/shared/env_loader.py new file mode 100644 index 0000000..6f9557e --- /dev/null +++ b/tests/pipeline_setup_v4/shared/env_loader.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +import os +from pathlib import Path + +from app.core.shared.config import load_workspace_env + + +def load_pipeline_setup_env(start_dir: str | Path | None = None) -> list[Path]: + base = Path(start_dir or Path.cwd()).resolve() + loaded = load_workspace_env(start_dir=base) + env_path = _find_v4_root(base) / ".env" + if env_path.is_file(): + _apply_env_file(env_path) + loaded.append(env_path) + return loaded + + +def _find_v4_root(base: Path) -> Path: + for directory in (base, *base.parents): + if directory.name == "pipeline_setup_v4" and (directory / "__init__.py").is_file(): + return directory + raise RuntimeError(f"Unable to locate tests/pipeline_setup_v4 root from: {base}") + + +def _apply_env_file(path: Path) -> None: + for raw_line in path.read_text(encoding="utf-8").splitlines(): + line = raw_line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, raw_value = line.split("=", 1) + name = key.removeprefix("export ").strip() + if name: + os.environ[name] = _normalize_value(raw_value.strip()) + + +def _normalize_value(value: str) -> str: + if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}: + return value[1:-1] + return value diff --git a/tests/unit_tests/agent/test_application_startup.py b/tests/unit_tests/agent/test_application_startup.py new file mode 100644 index 0000000..92d0244 --- /dev/null +++ b/tests/unit_tests/agent/test_application_startup.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +import app.core.application as application_module + + +def test_startup_keeps_backend_alive_when_database_bootstrap_fails(monkeypatch) -> None: + def fail_bootstrap(*_args, **_kwargs) -> None: + raise RuntimeError("db down") + + monkeypatch.setattr(application_module, "bootstrap_database", fail_bootstrap) + + app = application_module.ModularApplication() + + app.startup() + + health = app.health_payload() + assert health["status"] == "degraded" + assert health["reason"] == "database_unavailable" + assert "db down" in health["details"] diff --git a/tests/unit_tests/agent/test_gigachat_client_retry.py b/tests/unit_tests/agent/test_gigachat_client_retry.py index 288dc46..ea1e737 100644 --- a/tests/unit_tests/agent/test_gigachat_client_retry.py +++ b/tests/unit_tests/agent/test_gigachat_client_retry.py @@ -1,7 +1,7 @@ import requests -from app.modules.shared.gigachat.client import GigaChatClient -from app.modules.shared.gigachat.settings import GigaChatSettings +from app.core.shared.gigachat.client import GigaChatClient +from app.core.shared.gigachat.settings import GigaChatSettings class _FakeTokenProvider: diff --git a/tests/unit_tests/agent/test_llm_service_logging.py b/tests/unit_tests/agent/test_llm_service_logging.py index 6cb3fa3..090a172 100644 --- a/tests/unit_tests/agent/test_llm_service_logging.py +++ b/tests/unit_tests/agent/test_llm_service_logging.py @@ -1,6 +1,6 @@ import logging -from app.modules.agent.llm.service import AgentLlmService +from app.core.agent.llm.service import AgentLlmService class _FakeClient: @@ -19,7 +19,7 @@ class _FakePrompts: def test_llm_service_logs_input_and_output_for_graph_context(caplog) -> None: service = AgentLlmService(_FakeClient(), _FakePrompts()) - with caplog.at_level(logging.WARNING, logger="app.modules.agent.llm.service"): + with caplog.at_level(logging.WARNING, logger="app.core.agent.llm.service"): result = service.generate("general_answer", "User input", log_context="graph.default.answer") assert result == "LLM output" diff --git a/tests/unit_tests/agent/test_logging_setup.py b/tests/unit_tests/agent/test_logging_setup.py index 0521b56..0c9d1e1 100644 --- a/tests/unit_tests/agent/test_logging_setup.py +++ b/tests/unit_tests/agent/test_logging_setup.py @@ -1,6 +1,6 @@ import logging -from app.core.logging_setup import ScrubbingFormatter +from app.infra.logging_setup import ScrubbingFormatter def test_scrubbing_formatter_redacts_identifiers_and_adds_blank_line() -> None: diff --git a/tests/unit_tests/agent/test_repo_webhook_service.py b/tests/unit_tests/agent/test_repo_webhook_service.py deleted file mode 100644 index fe64d28..0000000 --- a/tests/unit_tests/agent/test_repo_webhook_service.py +++ /dev/null @@ -1,98 +0,0 @@ -from __future__ import annotations - -from app.modules.rag.webhook_service import RepoWebhookService - - -class FakeStoryWriter: - def __init__(self) -> None: - self.calls: list[dict] = [] - - def record_story_commit(self, **kwargs) -> None: - self.calls.append(kwargs) - - -class FakeCacheWriter: - def __init__(self) -> None: - self.calls: list[dict] = [] - - def record_repo_cache(self, **kwargs) -> None: - self.calls.append(kwargs) - - -def test_gitea_webhook_binds_story() -> None: - writer = FakeStoryWriter() - cache = FakeCacheWriter() - service = RepoWebhookService(writer, cache) - - result = service.process( - provider="gitea", - payload={ - "repository": {"full_name": "acme/proj"}, - "ref": "refs/heads/feature/AAAA-1234", - "pusher": {"username": "alice"}, - "commits": [ - { - "id": "abc123", - "message": "FEAT-1 update docs", - "added": ["docs/new.md"], - "modified": ["docs/api.md"], - "removed": [], - } - ], - }, - ) - - assert result["accepted"] is True - assert result["story_bound"] is True - assert result["story_id"] == "FEAT-1" - assert result["cache_recorded"] is True - assert len(writer.calls) == 1 - assert len(cache.calls) == 1 - assert writer.calls[0]["project_id"] == "acme/proj" - - -def test_webhook_without_story_id_is_non_fatal() -> None: - writer = FakeStoryWriter() - cache = FakeCacheWriter() - service = RepoWebhookService(writer, cache) - - result = service.process( - provider="bitbucket", - payload={ - "repository": {"full_name": "acme/proj"}, - "push": { - "changes": [ - { - "new": { - "name": "feature/no-story", - "target": {"hash": "abc123", "message": "update docs"}, - } - } - ] - }, - }, - ) - - assert result["accepted"] is True - assert result["story_bound"] is False - assert result["cache_recorded"] is True - assert len(cache.calls) == 1 - assert writer.calls == [] - - -def test_provider_autodetect_by_headers() -> None: - writer = FakeStoryWriter() - service = RepoWebhookService(writer) - - result = service.process( - headers={"X-Gitea-Event": "push"}, - payload={ - "repository": {"full_name": "acme/proj"}, - "ref": "refs/heads/feature/AAAA-1234", - "commits": [{"id": "abc123", "message": "AAAA-1234 update"}], - }, - ) - - assert result["accepted"] is True - assert result["story_bound"] is True - assert result["story_id"] == "AAAA-1234" diff --git a/tests/unit_tests/agent/test_session_service.py b/tests/unit_tests/agent/test_session_service.py new file mode 100644 index 0000000..20bc4d7 --- /dev/null +++ b/tests/unit_tests/agent/test_session_service.py @@ -0,0 +1,26 @@ +from __future__ import annotations + +from app.infra.exceptions import AppError +from app.core.api.application.session_service import SessionService +from app.core.api.infrastructure.ids.session_id_factory import SessionIdFactory +from app.core.api.infrastructure.stores.in_memory_session_store import InMemorySessionStore +import pytest + + +def test_create_session_allows_binding_rag_at_creation_time() -> None: + service = SessionService(store=InMemorySessionStore(), ids=SessionIdFactory()) + + session = service.create("rag_123") + + assert session.session_id.startswith("as_") + assert session.active_rag_session_id == "rag_123" + assert session.messages == [] + assert session.conversation_state.turn_index == 0 + + +def test_get_missing_session_raises_not_found() -> None: + service = SessionService(store=InMemorySessionStore(), ids=SessionIdFactory()) + with pytest.raises(AppError) as exc: + service.get("missing-session") + + assert exc.value.code == "session_not_found" diff --git a/tests/unit_tests/agent/test_v2_evidence_ranking.py b/tests/unit_tests/agent/test_v2_evidence_ranking.py new file mode 100644 index 0000000..c1032da --- /dev/null +++ b/tests/unit_tests/agent/test_v2_evidence_ranking.py @@ -0,0 +1,80 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.evidence.assembler import DocsEvidenceAssembler +from app.core.agent.processes.v2.models import V2Domain, V2Intent, V2RouteAnchors, V2RouteResult, V2Subintent + + +def _route(*, hints: list[str], terms: list[str], subintent: str = V2Subintent.SUMMARY) -> V2RouteResult: + return V2RouteResult( + routing_domain=V2Domain.DOCS, + intent=V2Intent.DOC_EXPLAIN, + subintent=subintent, + user_query="q", + normalized_query="q", + target_terms=terms, + anchors=V2RouteAnchors(target_doc_hints=hints, endpoint_paths=["/health"] if "/health" in terms else []), + ) + + +def test_target_doc_hint_is_hard_boosted_over_readme() -> None: + rows = [ + { + "path": "docs/README.md", + "title": "README", + "content": "", + "layer": "D1_DOCUMENT_CATALOG", + "metadata": {"summary_text": "index", "document_id": "docs.readme"}, + }, + { + "path": "docs/api/health-endpoint.md", + "title": "Health endpoint", + "content": "", + "layer": "D1_DOCUMENT_CATALOG", + "metadata": {"summary_text": "health summary", "document_id": "api.health"}, + }, + ] + route = _route( + hints=["docs/api/health-endpoint.md"], + terms=["/health", "health"], + ) + + docs = DocsEvidenceAssembler().assemble_summaries(rows, route) + + assert docs[0].path == "docs/api/health-endpoint.md" + assert docs[0].score_breakdown["target_doc_boost"] >= 1000 + + +def test_find_files_prefers_exact_path_match() -> None: + rows = [ + { + "path": "docs/architecture/telegram-notify-app-overview.md", + "title": "Overview", + "content": "overview", + "layer": "D1_DOCUMENT_CATALOG", + "metadata": {"document_id": "arch"}, + }, + { + "path": "docs/domains/runtime-health-entity.md", + "title": "Runtime health", + "content": "runtime health", + "layer": "D1_DOCUMENT_CATALOG", + "metadata": {"document_id": "domain.runtime"}, + }, + ] + route = V2RouteResult( + routing_domain=V2Domain.DOCS, + intent=V2Intent.DOC_EXPLAIN, + subintent=V2Subintent.FIND_FILES, + user_query="Где находится runtime health?", + normalized_query="Где находится runtime health?", + target_terms=["runtime", "health"], + anchors=V2RouteAnchors( + target_doc_hints=["docs/domains/runtime-health-entity.md"], + matched_aliases=["runtime-health-entity"], + ), + ) + + files = DocsEvidenceAssembler().assemble_files(rows, route) + + assert files[0].path == "docs/domains/runtime-health-entity.md" + assert files[0].match_reason in {"exact_path", "alias_match"} diff --git a/tests/unit_tests/agent/test_v2_intent_router.py b/tests/unit_tests/agent/test_v2_intent_router.py new file mode 100644 index 0000000..d73f0c5 --- /dev/null +++ b/tests/unit_tests/agent/test_v2_intent_router.py @@ -0,0 +1,98 @@ +from __future__ import annotations + +import json + +from app.core.agent.processes.v2 import V2IntentRouter + + +class FakeLlm: + def __init__(self, response: str, *, fail: bool = False) -> None: + self.response = response + self.fail = fail + self.calls: list[tuple[str, str]] = [] + + def generate(self, prompt_name: str, user_input: str, **_kwargs) -> str: + self.calls.append((prompt_name, user_input)) + if self.fail: + raise RuntimeError("llm unavailable") + return self.response + + +def _llm_response(domain: str, intent: str, subintent: str, *, confidence: float = 0.9, reason: str = "ok") -> str: + return json.dumps( + { + "routing_domain": domain, + "intent": intent, + "subintent": subintent, + "confidence": confidence, + "reason_short": reason, + }, + ensure_ascii=False, + ) + + +def test_router_uses_llm_as_default_selector() -> None: + llm = FakeLlm(_llm_response("DOCS", "DOC_EXPLAIN", "FIND_FILES", reason="file request")) + + result = V2IntentRouter(llm=llm).route("В каком файле описан RuntimeHealth?") + + assert result.subintent == "FIND_FILES" + assert result.routing_mode == "llm_default" + assert result.llm_router_used is True + assert result.confidence > 0.8 + assert len(llm.calls) == 1 + + +def test_router_falls_back_when_llm_returns_invalid_enum() -> None: + llm = FakeLlm(_llm_response("DOCS", "DOC_EXPLAIN", "MADE_UP")) + + result = V2IntentRouter(llm=llm).route("В каком документе описан runtime health?") + + assert result.routing_domain == "DOCS" + assert result.subintent == "FIND_FILES" + assert result.routing_mode == "llm_fallback" + assert result.llm_router_used is True + + +def test_router_falls_back_to_general_summary_when_llm_fails() -> None: + result = V2IntentRouter(llm=FakeLlm("{}", fail=True)).route("Расскажи что важно") + + assert result.subintent == "SUMMARY" + assert result.routing_mode == "llm_fallback" + assert result.llm_router_used is True + assert result.intent == "GENERAL_QA" + + +def test_router_keeps_endpoint_anchor_and_excludes_it_from_file_names() -> None: + result = V2IntentRouter(llm=FakeLlm(_llm_response("DOCS", "DOC_EXPLAIN", "SUMMARY"))).route( + "Что делает endpoint /send?" + ) + + assert result.anchors.endpoint_paths == ["/send"] + assert "/send" not in result.anchors.file_names + + +def test_router_cleans_target_terms_with_lowercase_rules() -> None: + result = V2IntentRouter(llm=FakeLlm(_llm_response("DOCS", "DOC_EXPLAIN", "FIND_FILES"))).route( + "Покажи где описан RuntimeHealth и /health" + ) + + assert "покажи" not in result.target_terms + assert "где" not in result.target_terms + assert "runtimehealth" in result.target_terms + assert "/health" in result.target_terms + + +def test_router_detects_doc_like_file_names_only() -> None: + result = V2IntentRouter(llm=FakeLlm(_llm_response("DOCS", "DOC_EXPLAIN", "FIND_FILES"))).route( + "Покажи document docs/architecture/overview.md и runtime_health" + ) + + assert "docs/architecture/overview.md" in result.anchors.file_names + assert "runtime_health" not in result.anchors.file_names + + +def test_router_reduces_confidence_for_short_vague_query() -> None: + result = V2IntentRouter(llm=FakeLlm(_llm_response("GENERAL", "GENERAL_QA", "SUMMARY", confidence=0.8))).route("Что это?") + + assert result.confidence < 0.8 diff --git a/tests/unit_tests/agent/test_v2_intent_router_extraction.py b/tests/unit_tests/agent/test_v2_intent_router_extraction.py new file mode 100644 index 0000000..27b60e6 --- /dev/null +++ b/tests/unit_tests/agent/test_v2_intent_router_extraction.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.intent_router.modules.anchors import V2AnchorExtractor +from app.core.agent.processes.v2.intent_router.modules.target_terms import V2TargetTermsExtractor + + +def test_target_terms_keeps_only_endpoint_for_short_explain_query() -> None: + analysis = V2TargetTermsExtractor().extract("Кратко объясни как работает /health") + + assert analysis.target_terms == ["/health"] + + +def test_target_terms_keeps_domain_terms_without_question_words() -> None: + analysis = V2TargetTermsExtractor().extract("Что делает runtime health") + + assert analysis.target_terms == ["runtime", "health"] + + +def test_target_terms_keeps_only_endpoint_for_file_marker_query() -> None: + analysis = V2TargetTermsExtractor().extract("В каком файле описан /health?") + + assert analysis.target_terms == ["/health"] + + +def test_target_terms_drops_noisy_filler_words() -> None: + analysis = V2TargetTermsExtractor().extract("Где там дока про health, покажи плз") + + assert analysis.target_terms == ["health"] + + +def test_target_terms_keeps_api_and_explicit_endpoint() -> None: + analysis = V2TargetTermsExtractor().extract("Нужен краткий док-саммари по api /send") + + assert analysis.target_terms == ["/send", "api"] + + +def test_target_terms_keeps_identifier_like_token() -> None: + analysis = V2TargetTermsExtractor().extract("Что делает telegram_notify?") + + assert analysis.target_terms == ["telegram_notify"] + + +def test_target_terms_rejects_broken_path_like_token() -> None: + analysis = V2TargetTermsExtractor().extract("Опиши /actions/{action") + + assert analysis.target_terms == [] + + +def test_file_names_accepts_real_doc_path() -> None: + terms = V2TargetTermsExtractor().extract("docs/api/health.md") + anchors = V2AnchorExtractor().extract("docs/api/health.md", terms).anchors + + assert anchors.file_names == ["docs/api/health.md"] + + +def test_file_names_rejects_endpoint_path() -> None: + terms = V2TargetTermsExtractor().extract("/health") + anchors = V2AnchorExtractor().extract("/health", terms).anchors + + assert anchors.file_names == [] + + +def test_file_names_rejects_identifier_like_token() -> None: + terms = V2TargetTermsExtractor().extract("telegram_notify") + anchors = V2AnchorExtractor().extract("telegram_notify", terms).anchors + + assert anchors.file_names == [] diff --git a/tests/unit_tests/agent/test_v2_process.py b/tests/unit_tests/agent/test_v2_process.py new file mode 100644 index 0000000..bd97774 --- /dev/null +++ b/tests/unit_tests/agent/test_v2_process.py @@ -0,0 +1,286 @@ +from __future__ import annotations + +import asyncio +from dataclasses import dataclass + +from app.core.agent.processes.v2 import V2IntentRouter, V2Process +from app.core.agent.processes.v2.retrieval.target_doc_seeding import normalize_doc_path +from app.core.agent.processes.v2.evidence.assembler import DocsEvidenceAssembler +from app.core.agent.processes.v2.evidence.gate import DocsEvidenceGate +from app.core.agent.processes.v2.retrieval.policy_resolver import V2RetrievalPolicyResolver +from app.core.agent.runtime.execution_context import RuntimeExecutionContext +from app.core.api.domain.models.agent_request import AgentRequest +from app.core.api.domain.models.agent_session import AgentSession +from app.schemas.orchestration import RequestExecutionStatus + + +class FakePublisher: + async def publish_status(self, *_args, **_kwargs) -> None: + return None + + async def publish_user(self, *_args, **_kwargs) -> None: + return None + + +class FakeTrace: + def __init__(self) -> None: + self.events: list[tuple[str, str, dict | None]] = [] + + def module(self, _name: str) -> "FakeTrace": + return self + + def log(self, title, payload=None, **_kwargs) -> None: + self.events.append(("module", str(title), payload)) + + +class FakeLlm: + def __init__(self, answer: str) -> None: + self.answer = answer + self.calls: list[tuple[str, str]] = [] + + def generate(self, prompt_name: str, user_input: str, **_kwargs) -> str: + self.calls.append((prompt_name, user_input)) + return self.answer + + +@dataclass(slots=True) +class FakeRagAdapter: + """Имитирует сырые строки из RagSessionRetriever для summary / find_files.""" + + summary_rows: list[dict] + file_rows: list[dict] + + async def fetch_rows(self, _rag_session_id: str, _query_text: str, plan) -> list[dict]: + if "find_files" in str(plan.profile) or str(plan.profile) == "file_lookup": + return list(self.file_rows) + return list(self.summary_rows) + + async def fetch_exact_paths(self, _rag_session_id: str, *, paths: list[str], layers: list[str] | None = None) -> list[dict]: + pool = [*self.summary_rows, *self.file_rows] + want = {normalize_doc_path(p) for p in paths} + return [row for row in pool if normalize_doc_path(str(row.get("path") or "")) in want] + + async def fetch_chunks_by_path_substrings( + self, + _rag_session_id: str, + *, + path_needles: list[str], + layers: list[str] | None = None, + limit: int = 200, + ) -> list[dict]: + del layers, limit + pool = [*self.summary_rows, *self.file_rows] + return [row for row in pool if any(needle in str(row.get("path") or "") for needle in path_needles)] + + +_SUMMARY_ROWS = [ + { + "path": "docs/api/health.md", + "title": "Health endpoint", + "content": "", + "layer": "D1_DOCUMENT_CATALOG", + "metadata": { + "summary_text": "Endpoint /health возвращает агрегированный статус runtime.", + "document_id": "api.health", + "title": "Health endpoint", + }, + } +] + +_FILE_ROWS = [ + { + "path": "docs/domains/runtime-health.md", + "title": "Runtime health", + "layer": "D3_ENTITY_CATALOG", + "content": "x", + "metadata": { + "entity_name": "RuntimeHealth", + "document_id": "domain.runtime_health", + }, + } +] + + +def _v2_process(llm: FakeLlm, adapter: FakeRagAdapter, *, workflow_llm_enabled: bool = True) -> V2Process: + return V2Process( + llm=llm, + policy_resolver=V2RetrievalPolicyResolver(), + rag_adapter=adapter, + evidence_assembler=DocsEvidenceAssembler(), + evidence_gate=DocsEvidenceGate(), + router=V2IntentRouter(), + workflow_llm_enabled=workflow_llm_enabled, + ) + + +def _context(message: str, *, rag_session_id: str | None = "rag-1") -> RuntimeExecutionContext: + request = AgentRequest( + request_id="req-1", + session_id="sess-1", + message=message, + process_version="v2", + status=RequestExecutionStatus.RUNNING, + created_at=AgentRequest.create("req-x", "sess-x", "x", "v2").created_at, + ) + session = AgentSession.create("sess-1", rag_session_id) + return RuntimeExecutionContext( + request=request, + session=session, + publisher=FakePublisher(), + trace=FakeTrace(), + ) + + +def test_v2_process_runs_summary_flow() -> None: + llm = FakeLlm("Краткое объяснение по документации.") + adapter = FakeRagAdapter(summary_rows=_SUMMARY_ROWS, file_rows=[]) + process = _v2_process(llm, adapter) + + result = asyncio.run(process.run(_context("Объясни /health в документации"))) + + assert result.answer == "Краткое объяснение по документации." + assert llm.calls + assert "docs/api/health.md" in llm.calls[0][1] + + +def test_v2_process_runs_find_files_flow_without_llm() -> None: + llm = FakeLlm("should not be used") + adapter = FakeRagAdapter(summary_rows=[], file_rows=_FILE_ROWS) + process = _v2_process(llm, adapter) + + result = asyncio.run(process.run(_context("В каком файле описан RuntimeHealth?"))) + + assert "docs/domains/runtime-health.md" in result.answer + assert llm.calls == [] + + +def test_v2_process_find_files_uses_deterministic_gate_mode() -> None: + llm = FakeLlm("unused") + adapter = FakeRagAdapter(summary_rows=[], file_rows=_FILE_ROWS) + process = _v2_process(llm, adapter) + runtime = _context("В каком документе описан runtime health?") + + asyncio.run(process.run(runtime)) + + pipeline_events = [payload for _, title, payload in runtime.trace.events if title == "evidence_gate_checked"] + assert pipeline_events + assert pipeline_events[0]["answer_mode"] == "deterministic" + + +def test_v2_process_runs_grounded_general_summary_with_rag() -> None: + llm = FakeLlm("Grounded summary.") + adapter = FakeRagAdapter(summary_rows=_SUMMARY_ROWS, file_rows=[]) + process = _v2_process(llm, adapter) + + result = asyncio.run(process.run(_context("Что это за сервис?"))) + + assert result.answer == "Grounded summary." + assert llm.calls + assert "Опорные документы" in llm.calls[0][1] + + +def test_v2_process_returns_insufficiency_for_general_without_rag() -> None: + llm = FakeLlm("Общий ответ без обращения к документации.") + adapter = FakeRagAdapter(summary_rows=[], file_rows=[]) + process = _v2_process(llm, adapter) + + result = asyncio.run(process.run(_context("Что это за сервис?", rag_session_id=None))) + + assert "grounded summary" in result.answer + assert llm.calls == [] + + +def test_v2_process_requires_active_rag_session() -> None: + process = _v2_process(FakeLlm("unused"), FakeRagAdapter([], [])) + + result = asyncio.run(process.run(_context("Объясни /health в документации", rag_session_id=None))) + + assert "нужна активная RAG-сессия" in result.answer + + +def test_v2_router_detects_find_files_subintent() -> None: + result = V2IntentRouter().route("В каком файле описан RuntimeHealth?") + + assert result.subintent == "FIND_FILES" + assert "RuntimeHealth" in result.anchors.entity_names + assert "runtimehealth" in result.target_terms + + +def test_v2_process_logs_retrieved_rag_rows_in_trace() -> None: + llm = FakeLlm("Краткое объяснение по документации.") + adapter = FakeRagAdapter(summary_rows=_SUMMARY_ROWS, file_rows=[]) + process = _v2_process(llm, adapter) + runtime = _context("Объясни /health в документации") + + asyncio.run(process.run(runtime)) + + retrieval_events = [payload for _, title, payload in runtime.trace.events if title == "rag_rows_fetched"] + assert retrieval_events + payload = retrieval_events[0] or {} + rows = payload.get("rows") or [] + assert rows + assert rows[0]["path"] == "docs/api/health.md" + assert rows[0]["layer"] == "D1_DOCUMENT_CATALOG" + assert rows[0]["document_id"] == "api.health" + + +def test_v2_process_logs_pipeline_steps() -> None: + llm = FakeLlm("Краткое объяснение по документации.") + adapter = FakeRagAdapter(summary_rows=_SUMMARY_ROWS, file_rows=[]) + process = _v2_process(llm, adapter) + runtime = _context("Что делает endpoint /health?") + + asyncio.run(process.run(runtime)) + + pipeline_titles = [title for _, title, _ in runtime.trace.events] + assert "router_resolved" in pipeline_titles + assert "anchors_extracted" in pipeline_titles + assert "retrieval_profile_selected" in pipeline_titles + assert "retrieval_executed" in pipeline_titles + assert "evidence_assembled" in pipeline_titles + assert "evidence_gate_checked" in pipeline_titles + assert "answer_generated" in pipeline_titles + + +def test_v2_process_blocks_generic_docs_answer_without_target_doc() -> None: + llm = FakeLlm("галлюцинация") + adapter = FakeRagAdapter( + summary_rows=[ + { + "path": "docs/README.md", + "title": "README", + "content": "", + "layer": "D1_DOCUMENT_CATALOG", + "metadata": {"summary_text": "Общий индекс документации.", "document_id": "docs.readme"}, + } + ], + file_rows=[], + ) + process = _v2_process(llm, adapter) + + result = asyncio.run(process.run(_context("Что делает endpoint /send?"))) + + assert "не найден целевой документ" in result.answer + assert llm.calls == [] + + +def test_v2_process_can_disable_workflow_llm_for_docs_summary() -> None: + llm = FakeLlm("should not be used") + adapter = FakeRagAdapter(summary_rows=_SUMMARY_ROWS, file_rows=[]) + process = _v2_process(llm, adapter, workflow_llm_enabled=False) + + result = asyncio.run(process.run(_context("Объясни /health в документации"))) + + assert "Endpoint /health возвращает агрегированный статус runtime." in result.answer + assert llm.calls == [] + + +def test_v2_process_can_disable_workflow_llm_for_general_summary() -> None: + llm = FakeLlm("should not be used") + adapter = FakeRagAdapter(summary_rows=_SUMMARY_ROWS, file_rows=[]) + process = _v2_process(llm, adapter, workflow_llm_enabled=False) + + result = asyncio.run(process.run(_context("Что это за сервис?"))) + + assert "агрегированный статус runtime" in result.answer + assert llm.calls == [] diff --git a/tests/unit_tests/agent/test_v2_retrieval_policy.py b/tests/unit_tests/agent/test_v2_retrieval_policy.py new file mode 100644 index 0000000..ab25423 --- /dev/null +++ b/tests/unit_tests/agent/test_v2_retrieval_policy.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +from app.core.agent.processes.v2.models import V2Domain, V2Intent, V2RouteAnchors, V2RouteResult, V2Subintent +from app.core.agent.processes.v2.retrieval.policy_resolver import V2RetrievalPolicyResolver + + +def _route(*, hints: list[str], endpoint_paths: list[str] | None = None, subintent: str = "SUMMARY", intent: str = "DOC_EXPLAIN") -> V2RouteResult: + return V2RouteResult( + routing_domain=V2Domain.DOCS if intent == V2Intent.DOC_EXPLAIN else V2Domain.GENERAL, + intent=intent, + subintent=subintent, + user_query="q", + normalized_query="q", + anchors=V2RouteAnchors(target_doc_hints=hints, endpoint_paths=endpoint_paths or []), + ) + + +def test_policy_prefers_api_docs_for_endpoint_queries() -> None: + plan = V2RetrievalPolicyResolver().resolve( + _route(hints=["docs/api/health-endpoint.md"], endpoint_paths=["/health"]) + ) + + assert plan.profile == "docs_summary_api_endpoint" + assert plan.filters["path_prefixes"] == ["docs/api/", "docs/architecture/", "docs/"] + assert plan.filters["prefer_path_prefixes"][0] == "docs/api/" + + +def test_policy_prefers_logic_docs_for_logic_queries() -> None: + plan = V2RetrievalPolicyResolver().resolve(_route(hints=["docs/logic/telegram-notification-loop.md"])) + + assert plan.profile == "docs_summary_logic_flow" + assert plan.filters["prefer_path_prefixes"][0] == "docs/logic/" + + +def test_policy_uses_deterministic_find_files_profile() -> None: + plan = V2RetrievalPolicyResolver().resolve( + _route(hints=["docs/api/health-endpoint.md"], endpoint_paths=["/health"], subintent=V2Subintent.FIND_FILES) + ) + + assert plan.profile == "file_lookup" + assert plan.layers == ["D1_DOCUMENT_CATALOG", "D3_ENTITY_CATALOG"] + assert "health-endpoint.md" in plan.filters["prefer_like_patterns"][0] + + +def test_policy_uses_grounded_general_profile() -> None: + plan = V2RetrievalPolicyResolver().resolve(_route(hints=[], intent=V2Intent.GENERAL_QA)) + + assert plan.profile == "general_qa_grounded_summary" + assert plan.filters["prefer_path_prefixes"][0] == "docs/architecture/" diff --git a/tests/unit_tests/api/test_route_map.py b/tests/unit_tests/api/test_route_map.py new file mode 100644 index 0000000..5e28a2e --- /dev/null +++ b/tests/unit_tests/api/test_route_map.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +from fastapi import FastAPI + +from app.main import create_app + + +def _route_map(app: FastAPI) -> set[tuple[str, str]]: + routes: set[tuple[str, str]] = set() + for route in app.routes: + methods = getattr(route, "methods", set()) or set() + for method in methods: + if method in {"HEAD", "OPTIONS"}: + continue + routes.add((method, route.path)) + return routes + + +def test_route_map_exposes_current_api_and_hides_legacy_index_routes() -> None: + app = create_app() + route_map = _route_map(app) + + assert ("POST", "/api/agent/sessions") in route_map + assert ("POST", "/api/agent/requests") in route_map + assert ("GET", "/api/agent/requests/{request_id}") in route_map + assert ("GET", "/api/agent/streams/{request_id}") in route_map + + assert ("GET", "/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}") in route_map + assert ("GET", "/api/rag/sessions/{rag_session_id}/jobs/{index_job_id}/events") in route_map + + assert ("POST", "/api/index/snapshot") not in route_map + assert ("POST", "/api/index/changes") not in route_map + assert ("GET", "/api/index/jobs/{index_job_id}") not in route_map + assert ("GET", "/api/index/jobs/{index_job_id}/events") not in route_map + assert ("POST", "/internal/rag/index/snapshot") not in route_map + assert ("POST", "/internal/rag/index/changes") not in route_map + assert ("GET", "/internal/rag/index/jobs/{index_job_id}") not in route_map + assert ("POST", "/internal/rag/retrieve") not in route_map + assert ("POST", "/internal/rag-repo/webhook") not in route_map + assert ("POST", "/api/agent/sessions/{session_id}/rag") not in route_map + assert ("POST", "/api/agent/sessions/{session_id}/reset") not in route_map + assert ("POST", "/api/rag/sessions") not in route_map + assert ("POST", "/api/rag/sessions/{rag_session_id}/changes") not in route_map diff --git a/tests/unit_tests/rag/asserts_intent_router.py b/tests/unit_tests/rag/asserts_intent_router.py index 97efbac..c72bce6 100644 --- a/tests/unit_tests/rag/asserts_intent_router.py +++ b/tests/unit_tests/rag/asserts_intent_router.py @@ -2,7 +2,7 @@ from __future__ import annotations import re -from app.modules.agent.intent_router_v2.models import IntentRouterResult +from app.core.agent.intent_router.models import IntentRouterResult def assert_intent(out: IntentRouterResult, expected: str) -> None: @@ -57,7 +57,7 @@ def assert_domain_layer_prefixes(out: IntentRouterResult) -> None: prefixes = {layer.layer_id[0] for layer in out.retrieval_spec.layer_queries if layer.layer_id} if out.retrieval_spec.domains == ["CODE"]: assert prefixes <= {"C"} - elif out.retrieval_spec.domains == ["DOCS"]: + elif out.retrieval_spec.domains in (["DOCS"], ["GENERAL"]): assert prefixes <= {"D"} else: assert prefixes <= {"C", "D"} diff --git a/tests/unit_tests/rag/intent_router_testkit.py b/tests/unit_tests/rag/intent_router_testkit.py index 52cbb51..26dcb8c 100644 --- a/tests/unit_tests/rag/intent_router_testkit.py +++ b/tests/unit_tests/rag/intent_router_testkit.py @@ -2,26 +2,22 @@ from __future__ import annotations import json -from app.modules.rag.contracts.enums import RagLayer -from app.modules.agent.intent_router_v2 import ConversationState, IntentRouterV2, RepoContext +from app.core.rag.contracts.enums import RagLayer +from app.core.agent.intent_router import ConversationState, IntentRouterV2, RepoContext def repo_context() -> RepoContext: return RepoContext( languages=["python"], - available_domains=["CODE", "DOCS"], + available_domains=["DOCS", "GENERAL"], available_layers=[ - RagLayer.CODE_ENTRYPOINTS, - RagLayer.CODE_SYMBOL_CATALOG, - RagLayer.CODE_DEPENDENCY_GRAPH, - RagLayer.CODE_SEMANTIC_ROLES, - RagLayer.CODE_SOURCE_CHUNKS, RagLayer.DOCS_DOC_CHUNKS, RagLayer.DOCS_DOCUMENT_CATALOG, RagLayer.DOCS_FACT_INDEX, RagLayer.DOCS_ENTITY_CATALOG, RagLayer.DOCS_WORKFLOW_INDEX, RagLayer.DOCS_RELATION_GRAPH, + RagLayer.DOCS_INTEGRATION_INDEX, ], ) diff --git a/tests/unit_tests/rag/test_code_indexing_pipeline.py b/tests/unit_tests/rag/test_code_indexing_pipeline.py index 0b39e29..74bdc74 100644 --- a/tests/unit_tests/rag/test_code_indexing_pipeline.py +++ b/tests/unit_tests/rag/test_code_indexing_pipeline.py @@ -1,5 +1,5 @@ -from app.modules.rag.contracts.enums import RagLayer -from app.modules.rag.indexing.code.pipeline import CodeIndexingPipeline +from app.core.rag.contracts.enums import RagLayer +from app.core.rag.indexing.code.pipeline import CodeIndexingPipeline def test_code_pipeline_builds_source_symbols_edges_and_entrypoints() -> None: diff --git a/tests/unit_tests/rag/test_docs_indexing_pipeline.py b/tests/unit_tests/rag/test_docs_indexing_pipeline.py index 98f3bcb..e551e23 100644 --- a/tests/unit_tests/rag/test_docs_indexing_pipeline.py +++ b/tests/unit_tests/rag/test_docs_indexing_pipeline.py @@ -1,15 +1,18 @@ -from app.modules.rag.contracts.enums import RagLayer -from app.modules.rag.indexing.docs.pipeline import DocsIndexingPipeline +from app.core.rag.contracts.enums import RagLayer +from app.core.rag.indexing.docs.pipeline import DocsIndexingPipeline -def test_docs_pipeline_builds_new_d0_to_d5_layers() -> None: +def test_docs_pipeline_builds_docs_layers_from_modern_markdown_structure() -> None: pipeline = DocsIndexingPipeline() content = """--- id: api.billing.create_invoice type: api_method +doc_type: api_method name: create_invoice title: Create Invoice API module: billing +domain: billing +sub_domain: invoices layer: application status: draft updated_at: 2026-03-23 @@ -17,21 +20,26 @@ tags: [billing, api] entities: [Invoice] parent: billing_api children: [] +related_docs: [api.billing.validate_invoice] links: - - type: related_api - target: api.billing.validate_invoice + called_by: + - ui.billing.invoice_form + uses_logic: + - logic.billing.invoice_validation --- -# Summary +# Create Invoice API + +## Summary Creates an invoice in billing. -# Details +## Details -## Описание +### Описание Создает счет на оплату. -## Сценарий +### Сценарий **Название:** Create invoice @@ -55,7 +63,12 @@ Create invoice **Постусловие:** - Invoice is created. -## Контракт +### Контракт + +#### Метаданные вызова +- Method: POST +- Auth: USER +- Idempotency: false ### Входные параметры @@ -69,7 +82,22 @@ Create invoice | --- | --- | --- | | invoice_id | string | yes | -## Ошибки +### Интеграции + +#### Billing DB +- target: db.billing.invoices +- target_type: db +- direction: outbound +- interaction: writes +- via: invoice repository +- purpose: persist created invoices +- details: + - transaction: required + - tables: + - invoices + - invoice_items + +### Ошибки | status | error | client action | | --- | --- | --- | @@ -89,10 +117,14 @@ Create invoice assert RagLayer.DOCS_ENTITY_CATALOG in layers assert RagLayer.DOCS_WORKFLOW_INDEX in layers assert RagLayer.DOCS_RELATION_GRAPH in layers + assert RagLayer.DOCS_INTEGRATION_INDEX in layers catalog_doc = next(doc for doc in docs if doc.layer == RagLayer.DOCS_DOCUMENT_CATALOG) assert catalog_doc.metadata["document_id"] == "api.billing.create_invoice" assert catalog_doc.metadata["module"] == "billing" + assert catalog_doc.metadata["domain"] == "billing" + assert catalog_doc.metadata["subdomain"] == "invoices" + assert catalog_doc.metadata["summary_text"] == "Creates an invoice in billing." fact_texts = [doc.text for doc in docs if doc.layer == RagLayer.DOCS_FACT_INDEX] assert any("has_field amount" in text for text in fact_texts) @@ -108,6 +140,16 @@ Create invoice relation_targets = [doc.metadata["target_id"] for doc in docs if doc.layer == RagLayer.DOCS_RELATION_GRAPH] assert "billing_api" in relation_targets assert "api.billing.validate_invoice" in relation_targets + assert "logic.billing.invoice_validation" in relation_targets + assert "Invoice" in relation_targets chunk_doc = next(doc for doc in docs if doc.layer == RagLayer.DOCS_DOC_CHUNKS) assert chunk_doc.metadata["section_path"] + assert chunk_doc.metadata["artifact_type"] == "DOCS" + assert chunk_doc.metadata["domain"] == "billing" + assert chunk_doc.metadata["subdomain"] == "invoices" + + integration_doc = next(doc for doc in docs if doc.layer == RagLayer.DOCS_INTEGRATION_INDEX) + assert integration_doc.metadata["target"] == "db.billing.invoices" + assert integration_doc.metadata["target_type"] == "db" + assert integration_doc.metadata["details"]["transaction"] == "required" diff --git a/tests/unit_tests/rag/test_docs_prompt_layer.py b/tests/unit_tests/rag/test_docs_prompt_layer.py index 370bd9b..3048174 100644 --- a/tests/unit_tests/rag/test_docs_prompt_layer.py +++ b/tests/unit_tests/rag/test_docs_prompt_layer.py @@ -1,11 +1,12 @@ from __future__ import annotations -from app.modules.agent.intent_router_v2 import IntentRouterV2 -from app.modules.agent.llm.prompt_loader import PromptLoader -from app.modules.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner -from app.modules.agent.runtime.docs_qa_pipeline.openapi_postprocessor import OpenAPIPostprocessor -from app.modules.agent.runtime.docs_qa_pipeline.prompt_payload_builder import DocsPromptPayloadBuilder -from app.modules.agent.runtime.steps.generation import RuntimePromptSelector +from app.core.agent.intent_router import IntentRouterV2 +from app.core.agent.llm.prompt_loader import PromptLoader +from app.core.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner +from app.core.agent.runtime.docs_qa_pipeline.openapi_postprocessor import OpenAPIPostprocessor +from app.core.agent.runtime.docs_qa_pipeline.prompt_payload_builder import DocsPromptPayloadBuilder +from app.core.agent.orchestration.processes.v2.prompt_payload_builder import V2PromptPayloadBuilder +from app.core.agent.runtime.steps.generation import RuntimePromptSelector from tests.docs_qa_eval.fixture_adapter import InMemoryDocsRetrievalAdapter from tests.unit_tests.rag.intent_router_testkit import repo_context @@ -43,7 +44,7 @@ def test_prompt_selector_uses_docs_prompts_only() -> None: def test_docs_prompt_payload_contains_required_contract() -> None: builder = DocsPromptPayloadBuilder() - from app.modules.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle, OpenAPIResult + from app.core.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle, OpenAPIResult payload = builder.build( question="Объясни billing", @@ -52,6 +53,8 @@ def test_docs_prompt_payload_contains_required_contract() -> None: evidence_bundle=DocsEvidenceBundle( intent="DOCUMENTATION_EXPLAIN", sub_intent="COMPONENT_EXPLAIN", + primary_documents=[{"title": "Billing"}], + secondary_documents=[{"title": "Billing relation"}], documents=[{"title": "Billing"}], facts=[{"content": "Handles payments"}], relations=[{"title": "Billing -> Orders"}], @@ -62,12 +65,36 @@ def test_docs_prompt_payload_contains_required_contract() -> None: assert '"question": "Объясни billing"' in payload assert '"intent": "DOCUMENTATION_EXPLAIN"' in payload assert '"sub_intent": "COMPONENT_EXPLAIN"' in payload + assert '"primary_documents"' in payload + assert '"secondary_documents"' in payload assert '"documents"' in payload assert '"facts"' in payload assert '"relations"' in payload assert '"api_contract"' in payload +def test_v2_prompt_payload_accepts_api_method_mode_fields() -> None: + from app.core.agent.runtime.docs_qa_pipeline.models import DocsEvidenceBundle + + payload = V2PromptPayloadBuilder().build( + question="Как работает метод health?", + intent="DOCUMENTATION_EXPLAIN", + sub_intent="API_METHOD_EXPLAIN", + evidence_bundle=DocsEvidenceBundle(intent="DOCUMENTATION_EXPLAIN", sub_intent="API_METHOD_EXPLAIN"), + api_method_answer_mode="indirect", + target_endpoint_identity={ + "anchor": "health", + "normalized_path": "/health", + "normalized_doc_id": "api.health_endpoint", + }, + direct_api_spec_found=False, + ) + + assert '"api_method_answer_mode": "indirect"' in payload + assert '"normalized_doc_id": "api.health_endpoint"' in payload + assert '"direct_api_spec_found": false' in payload + + def test_openapi_postprocessor_requires_paths_for_full_spec() -> None: validator = OpenAPIPostprocessor() diff --git a/tests/unit_tests/rag/test_docs_qa_pipeline.py b/tests/unit_tests/rag/test_docs_qa_pipeline.py index 82c96ac..10766e3 100644 --- a/tests/unit_tests/rag/test_docs_qa_pipeline.py +++ b/tests/unit_tests/rag/test_docs_qa_pipeline.py @@ -1,7 +1,7 @@ from __future__ import annotations -from app.modules.agent.intent_router_v2 import IntentRouterV2 -from app.modules.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner +from app.core.agent.intent_router import IntentRouterV2 +from app.core.agent.runtime.docs_qa_pipeline import DocsQAPipelineRunner, DocsTaskPlanner from tests.docs_qa_eval.fixture_adapter import InMemoryDocsRetrievalAdapter from tests.unit_tests.rag.intent_router_testkit import repo_context @@ -135,6 +135,52 @@ def test_openapi_partial_contract_returns_partial_mode() -> None: assert "/orders" in result.answer +def test_docs_pipeline_accepts_precomputed_task_plan_without_rerouting() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/api/health.md", + "title": "GET /health", + "content": "/health returns runtime and component statuses.", + "metadata": {"document_id": "api.health_endpoint", "type": "api_method", "endpoint": "/health"}, + }, + { + "layer": "D2_FACT_INDEX", + "path": "docs/api/health.md", + "title": "api.health_endpoint:response", + "content": "Returns health summary and component diagnostics.", + "metadata": {"subject_id": "api.health_endpoint", "type": "api_method"}, + }, + ] + route_result = IntentRouterV2().route( + "Объясни API метод /health", + repo_context=repo_context(), + ) + task_plan = DocsTaskPlanner().plan( + "Объясни API метод /health", + "docs-session", + route_result=route_result, + ) + + class FailingRouter: + def route(self, *_args, **_kwargs): + raise AssertionError("runner should use the precomputed task plan") + + runner = DocsQAPipelineRunner(FailingRouter(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context()) + + result = runner.run( + "Объясни API метод /health", + "docs-session", + mode="pre_llm_only", + task_plan=task_plan, + ) + + assert result.router_result.intent == "DOCUMENTATION_EXPLAIN" + assert result.router_result.query_plan.sub_intent == "API_METHOD_EXPLAIN" + assert result.diagnostics.selected_primary_documents == ["api.health_endpoint"] + assert result.diagnostics.gate_decision == "allow_exact" + + def test_pre_llm_mode_returns_diagnostic_only_without_answer_generation() -> None: rows = [ { @@ -172,7 +218,7 @@ def test_pre_llm_mode_detects_path_anchor_candidates() -> None: assert "/health" in result.diagnostics.query_anchor_candidates assert "/health" in result.diagnostics.resolved_anchor_candidates - assert result.diagnostics.planned_layers == ["D2_FACT_INDEX", "D4_WORKFLOW_INDEX", "D1_DOCUMENT_CATALOG", "D0_DOC_CHUNKS"] + assert result.diagnostics.planned_layers == ["D1_DOCUMENT_CATALOG", "D2_FACT_INDEX", "D0_DOC_CHUNKS", "D4_WORKFLOW_INDEX"] assert set(result.diagnostics.executed_layers) == {"D1_DOCUMENT_CATALOG", "D2_FACT_INDEX", "D4_WORKFLOW_INDEX", "D0_DOC_CHUNKS"} @@ -318,8 +364,311 @@ def test_openapi_request_fragment_uses_fragment_aware_gate() -> None: assert result.answer_mode in {"ready", "ready_partial"} assert result.answer assert "type: object" in result.answer - assert "message:" in result.answer - assert "chat_id:" in result.answer + + +def test_api_method_explain_prefers_api_method_primary_doc() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/domain/runtime-health.md", + "title": "Сущность runtime health", + "content": "Runtime health describes overall service health.", + "metadata": {"document_id": "domain.runtime_health", "type": "domain_entity"}, + }, + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/api/health.md", + "title": "GET /health", + "content": "/health returns runtime and component statuses.", + "metadata": {"document_id": "api.health_endpoint", "type": "api_method", "endpoint": "/health"}, + }, + { + "layer": "D5_RELATION_GRAPH", + "path": "docs/domain/runtime-health.md", + "title": "Runtime health links", + "content": "runtime health used by health endpoint", + "metadata": {"document_id": "domain.runtime_health", "target_doc_id": "api.health_endpoint"}, + }, + ] + runner = DocsQAPipelineRunner(IntentRouterV2(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context()) + + result = runner.run("Что делает метод health?", "docs-session", mode="pre_llm_only") + + assert result.router_result.query_plan.sub_intent == "API_METHOD_EXPLAIN" + assert result.diagnostics.target_anchor in {"health", "/health"} + assert result.diagnostics.api_method_match_found is True + assert result.diagnostics.selected_primary_documents == ["api.health_endpoint"] + assert "api.health_endpoint" in result.diagnostics.primary_doc_candidates + assert result.diagnostics.evidence_gate_require_target_api_spec is True + assert result.diagnostics.evidence_gate_target_api_spec_found is True + assert result.answer_mode == "exact" + assert result.diagnostics.gate_decision == "allow_exact" + + +def test_api_method_explain_promotes_api_doc_via_links() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/domain/runtime-health.md", + "title": "Сущность runtime health", + "content": "Runtime health is the domain model for observability.", + "metadata": { + "document_id": "domain.runtime_health", + "type": "domain_entity", + "links": [{"target": "api.health_endpoint", "type": "used_by"}], + }, + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/api/health.md", + "title": "api.health_endpoint:Overview", + "content": "Endpoint /health returns overall runtime status and component diagnostics.", + "metadata": {"document_id": "api.health_endpoint", "type": "api_method", "endpoint": "/health"}, + }, + ] + runner = DocsQAPipelineRunner(IntentRouterV2(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context()) + + result = runner.run("Как работает health endpoint?", "docs-session", mode="pre_llm_only") + + assert result.diagnostics.promoted_via_links == ["api.health_endpoint"] + assert result.diagnostics.selected_primary_documents == ["api.health_endpoint"] + assert result.diagnostics.api_method_match_found is True + + +def test_api_method_explain_rejects_cross_endpoint_primary_candidates() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/documentation/api/control-actions-endpoint.md", + "title": "HTTP API /actions/{action}", + "content": "Endpoint for controlling actions.", + "metadata": { + "document_id": "api.control_actions_endpoint", + "type": "api_method", + "endpoint": "/actions/{action}", + }, + }, + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/documentation/api/health-endpoint.md", + "title": "HTTP API /health", + "content": "Health endpoint returns runtime health and component diagnostics.", + "metadata": { + "document_id": "api.health_endpoint", + "type": "api_method", + "endpoint": "/health", + }, + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/documentation/api/actions-endpoint.md", + "title": "api.control_actions_endpoint:Scenario", + "content": "The /actions/{action} endpoint triggers runtime actions.", + "metadata": { + "document_id": "api.control_actions_endpoint", + "type": "api_method", + "endpoint": "/actions/{action}", + }, + }, + ] + runner = DocsQAPipelineRunner(IntentRouterV2(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context()) + + result = runner.run("Как работает метод health?", "docs-session", mode="pre_llm_only") + + assert result.router_result.query_plan.sub_intent == "API_METHOD_EXPLAIN" + assert result.diagnostics.target_endpoint_identity["normalized_doc_id"] == "api.health_endpoint" + assert result.diagnostics.selected_primary_documents == ["api.health_endpoint"] + assert result.diagnostics.primary_api_documents_after_filter == ["api.health_endpoint"] + assert "api.control_actions_endpoint" in result.diagnostics.rejected_endpoint_candidates + assert result.diagnostics.cross_endpoint_leakage_detected is True + assert result.diagnostics.evidence_gate_target_api_spec_found is True + assert "api.control_actions_endpoint" not in result.diagnostics.selected_doc_ids + + +def test_api_method_explain_without_exact_target_returns_insufficiency() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/documentation/api/control-actions-endpoint.md", + "title": "HTTP API /actions/{action}", + "content": "Endpoint for controlling actions.", + "metadata": { + "document_id": "api.control_actions_endpoint", + "type": "api_method", + "endpoint": "/actions/{action}", + }, + }, + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/documentation/api/send-endpoint.md", + "title": "HTTP API /send", + "content": "Endpoint for sending messages.", + "metadata": { + "document_id": "api.send_message_endpoint", + "type": "api_method", + "endpoint": "/send", + }, + }, + ] + runner = DocsQAPipelineRunner(IntentRouterV2(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context()) + + result = runner.run("Что делает метод health?", "docs-session", mode="pre_llm_only") + + assert result.router_result.query_plan.sub_intent == "API_METHOD_EXPLAIN" + assert result.diagnostics.target_endpoint_identity["normalized_doc_id"] == "api.health_endpoint" + assert result.diagnostics.selected_primary_documents == [] + assert "api.control_actions_endpoint" in result.diagnostics.rejected_endpoint_candidates + assert "api.send_message_endpoint" in result.diagnostics.rejected_endpoint_candidates + assert result.diagnostics.target_api_spec_found_exact is False + assert result.diagnostics.evidence_gate_target_api_spec_found is False + assert result.diagnostics.gate_decision == "reject" + assert result.answer_mode == "insufficient" + assert "api.control_actions_endpoint" not in result.diagnostics.selected_doc_ids + assert "api.send_message_endpoint" not in result.diagnostics.selected_doc_ids + + +def test_api_method_explain_uses_indirect_mode_from_target_linked_docs() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/domain/runtime-health.md", + "title": "Runtime health", + "content": "Runtime health describes overall service state and component diagnostics.", + "metadata": { + "document_id": "domain.runtime_health", + "type": "domain_entity", + "links": [{"target": "api.health_endpoint", "type": "used_by"}], + }, + }, + ] + runner = DocsQAPipelineRunner(IntentRouterV2(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context()) + + result = runner.run("Как работает метод health?", "docs-session", mode="pre_llm_only") + + assert result.answer_mode == "indirect" + assert result.diagnostics.gate_decision == "allow_indirect" + assert result.diagnostics.raw_retrieval_non_empty is True + assert result.diagnostics.target_primary_context_non_empty is False + assert result.diagnostics.indirect_target_context_non_empty is True + assert result.diagnostics.graph_promotion_attempted is True + assert result.diagnostics.graph_promotion_hits == ["api.health_endpoint"] + assert result.diagnostics.promoted_target_loaded is False + assert result.diagnostics.materialization_failure_reason == "materialized_rows_empty" + assert result.diagnostics.final_primary_document_ids == [] + assert "domain.runtime_health" in result.diagnostics.final_secondary_document_ids + + +def test_api_method_explain_skips_llm_when_no_exact_or_indirect_context() -> None: + from tests.unit_tests.rag.test_docs_prompt_layer import FakeLlm + + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/documentation/api/send-endpoint.md", + "title": "HTTP API /send", + "content": "Endpoint for sending messages.", + "metadata": { + "document_id": "api.send_message_endpoint", + "type": "api_method", + "endpoint": "/send", + }, + } + ] + llm = FakeLlm("should not be called") + runner = DocsQAPipelineRunner(IntentRouterV2(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context(), llm=llm) + + result = runner.run("Что делает метод health?", "docs-session") + + assert llm.calls == [] + assert result.answer_mode == "insufficient" + assert result.diagnostics.llm_called is False + assert result.diagnostics.llm_call_reason == "no_exact_or_indirect_target_context" + assert result.diagnostics.gate_decision == "reject" + + +def test_api_method_explain_materializes_promoted_target_into_primary_context() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/domain/runtime-health.md", + "title": "Runtime health", + "content": "Runtime health describes service state and component diagnostics.", + "metadata": { + "document_id": "domain.runtime_health", + "type": "domain_entity", + "links": [{"target": "api.health_endpoint", "type": "used_by"}], + }, + } + ] + materialized_rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/api/health.md", + "title": "GET /health", + "content": "/health returns runtime and component statuses.", + "metadata": {"document_id": "api.health_endpoint", "type": "api_method", "endpoint": "/health"}, + }, + { + "layer": "D2_FACT_INDEX", + "path": "docs/api/health.md", + "title": "api.health_endpoint:response", + "content": "Returns health summary and component diagnostics.", + "metadata": {"subject_id": "api.health_endpoint", "type": "api_method"}, + }, + { + "layer": "D0_DOC_CHUNKS", + "path": "docs/api/health.md", + "title": "api.health_endpoint:Overview", + "content": "Endpoint /health returns overall runtime health.", + "metadata": {"document_id": "api.health_endpoint", "type": "api_method", "endpoint": "/health"}, + }, + ] + runner = DocsQAPipelineRunner( + IntentRouterV2(), + InMemoryDocsRetrievalAdapter(rows, materialized_rows=materialized_rows), + repo_context=repo_context(), + ) + + result = runner.run("Как работает метод health?", "docs-session", mode="pre_llm_only") + + assert result.answer_mode == "exact" + assert result.diagnostics.graph_promotion_hits == ["api.health_endpoint"] + assert result.diagnostics.graph_promotion_materialized == ["api.health_endpoint"] + assert result.diagnostics.promoted_target_loaded is True + assert result.diagnostics.promoted_target_chunks_loaded == 1 + assert result.diagnostics.promoted_target_facts_loaded == 1 + assert result.diagnostics.pinned_document_ids == ["api.health_endpoint"] + assert result.diagnostics.final_primary_document_ids == ["api.health_endpoint"] + assert "domain.runtime_health" in result.diagnostics.final_secondary_document_ids + assert result.diagnostics.materialized_target_primary_context_non_empty is True + assert result.diagnostics.gate_decision == "allow_exact" + + +def test_entity_question_does_not_prefer_api_method_primary_doc() -> None: + rows = [ + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/domain/runtime-health.md", + "title": "Сущность runtime health", + "content": "Runtime health describes service state.", + "metadata": {"document_id": "domain.runtime_health", "type": "domain_entity"}, + }, + { + "layer": "D1_DOCUMENT_CATALOG", + "path": "docs/api/health.md", + "title": "GET /health", + "content": "/health returns runtime status.", + "metadata": {"document_id": "api.health_endpoint", "type": "api_method", "endpoint": "/health"}, + }, + ] + runner = DocsQAPipelineRunner(IntentRouterV2(), InMemoryDocsRetrievalAdapter(rows), repo_context=repo_context()) + + result = runner.run("Что такое runtime health?", "docs-session", mode="pre_llm_only") + + assert result.router_result.query_plan.sub_intent == "ENTITY_EXPLAIN" + assert result.diagnostics.selected_primary_documents == [] + assert result.diagnostics.api_method_match_found is False + assert result.answer == "" def test_openapi_method_with_only_path_is_rejected() -> None: diff --git a/tests/unit_tests/rag/test_docs_retrieval_plan_mapping.py b/tests/unit_tests/rag/test_docs_retrieval_plan_mapping.py new file mode 100644 index 0000000..2462905 --- /dev/null +++ b/tests/unit_tests/rag/test_docs_retrieval_plan_mapping.py @@ -0,0 +1,32 @@ +import pytest + +from tests.unit_tests.rag.intent_router_testkit import run_sequence + +pytestmark = pytest.mark.intent_router + + +@pytest.mark.parametrize( + ("query", "plan_id", "primary_doc_types", "secondary_doc_types", "expected_filter_key", "expected_filter_value"), + [ + ("объясни /health", "docs_api_method_explain_v1", ["api_method"], ["logic_block", "domain_entity", "architecture_overview"], "endpoint_path", "/health"), + ("какие методы в notifications", "docs_list_api_methods_v1", ["api_method"], [], "domain_name", "notifications"), + ("найди документацию по telegram_delivery", "docs_find_documents_by_domain_v1", ["index_page", "architecture_overview", "api_method", "logic_block", "domain_entity"], [], "domain_name", "telegram_delivery"), + ("сгенерируй openapi по /send", "docs_generate_openapi_v1", ["api_method"], ["domain_entity", "logic_block"], "endpoint_path", "/send"), + ("как устроен сервис", "docs_general_docs_qa_v1", ["index_page", "architecture_overview"], ["logic_block", "domain_entity", "api_method"], "scope_level", "project"), + ], +) +def test_docs_retrieval_plan_contracts( + query: str, + plan_id: str, + primary_doc_types: list[str], + secondary_doc_types: list[str], + expected_filter_key: str, + expected_filter_value: str, +) -> None: + result = run_sequence([query])[0] + + assert result.retrieval_plan is not None + assert result.retrieval_plan.plan_id == plan_id + assert result.retrieval_plan.primary_doc_types == primary_doc_types + assert result.retrieval_plan.secondary_doc_types == secondary_doc_types + assert result.retrieval_plan.filters[expected_filter_key] == expected_filter_value diff --git a/tests/unit_tests/rag/test_explain_intent_builder.py b/tests/unit_tests/rag/test_explain_intent_builder.py index c76ea12..b50b86a 100644 --- a/tests/unit_tests/rag/test_explain_intent_builder.py +++ b/tests/unit_tests/rag/test_explain_intent_builder.py @@ -1,4 +1,4 @@ -from app.modules.agent.runtime.steps.explain.intent_builder import ExplainIntentBuilder +from app.core.agent.runtime.steps.explain.intent_builder import ExplainIntentBuilder def test_explain_intent_builder_extracts_route_symbol_and_file_hints() -> None: diff --git a/tests/unit_tests/rag/test_intent_router_e2e_flows.py b/tests/unit_tests/rag/test_intent_router_e2e_flows.py index 803e824..f4a3e1d 100644 --- a/tests/unit_tests/rag/test_intent_router_e2e_flows.py +++ b/tests/unit_tests/rag/test_intent_router_e2e_flows.py @@ -2,13 +2,12 @@ import os import pytest -from app.modules.agent.intent_router_v2.factory import GigaChatIntentRouterFactory -from app.modules.shared.env_loader import load_workspace_env +from app.core.agent.intent_router.factory import GigaChatIntentRouterFactory +from app.core.shared.config import load_workspace_env from tests.unit_tests.rag.asserts_intent_router import ( assert_domains, - assert_file_only_scope, assert_intent, - assert_test_policy, + assert_path_scope, ) from tests.unit_tests.rag.intent_router_testkit import run_sequence @@ -29,7 +28,7 @@ def test_e2e_path_carryover_flow() -> None: ] ) - assert_file_only_scope(first, "app/core/config.py") + assert_path_scope(first, "app/core/config.py", "app/core") assert "app/core/config.py" in second.retrieval_spec.filters.path_scope assert "app/core/config.py" in third.retrieval_spec.filters.path_scope second_file_anchors = [anchor.value for anchor in second.query_plan.anchors if anchor.type == "FILE_PATH" and anchor.source == "conversation_state"] @@ -39,7 +38,10 @@ def test_e2e_path_carryover_flow() -> None: assert any(anchor.type == "FILE_PATH" and anchor.source == "conversation_state" and anchor.span is None for anchor in third.query_plan.anchors) carried_symbols = [anchor.value for anchor in third.query_plan.anchors if anchor.type == "SYMBOL" and anchor.source == "conversation_state"] assert carried_symbols in ([], ["load_config"]) - assert third.query_plan.sub_intent == "EXPLAIN_LOCAL" + assert_intent(first, "GENERAL_QA") + assert_intent(second, "GENERAL_QA") + assert_intent(third, "GENERAL_QA") + assert third.query_plan.sub_intent == "GENERIC_QA" layer_ids = [item.layer_id for item in third.retrieval_spec.layer_queries] assert "C3_ENTRYPOINTS" not in layer_ids @@ -52,9 +54,9 @@ def test_e2e_docs_switch_from_code_topic() -> None: ] ) - assert_intent(first, "CODE_QA") + assert_intent(first, "DOCUMENTATION_EXPLAIN") assert_intent(second, "DOCUMENTATION_EXPLAIN") - assert second.conversation_mode == "SWITCH" + assert second.conversation_mode == "CONTINUE" assert_domains(second, ["DOCS"]) carried = [ anchor @@ -75,12 +77,10 @@ def test_e2e_tests_toggle_flow() -> None: ] ) - assert_intent(first, "CODE_QA") - assert_intent(second, "CODE_QA") - assert_test_policy(first, "INCLUDE") - assert_test_policy(second, "EXCLUDE") - assert first.query_plan.sub_intent == "FIND_TESTS" - assert second.query_plan.sub_intent == "EXPLAIN" + assert_intent(first, "GENERAL_QA") + assert_intent(second, "GENERAL_QA") + assert first.query_plan.sub_intent == "GENERIC_QA" + assert second.query_plan.sub_intent == "GENERIC_QA" assert "tests" in second.query_plan.negations assert not second.query_plan.expansions assert second.evidence_policy.require_flow is False @@ -94,9 +94,9 @@ def test_e2e_open_file_then_generic_next_steps_is_lightweight() -> None: ] ) - assert_file_only_scope(first, "app/core/config.py") - assert_file_only_scope(second, "app/core/config.py") - assert second.query_plan.sub_intent in {"EXPLAIN_LOCAL", "NEXT_STEPS"} + assert_path_scope(first, "app/core/config.py", "app/core") + assert_path_scope(second, "app/core/config.py", "app/core") + assert second.query_plan.sub_intent == "GENERIC_QA" layer_ids = [item.layer_id for item in second.retrieval_spec.layer_queries] assert "C3_ENTRYPOINTS" not in layer_ids assert second.evidence_policy.require_flow is False @@ -118,9 +118,9 @@ def test_intent_router_live_smoke_path_carryover() -> None: trace_label="intent-router-live", ) - assert_file_only_scope(first, "app/core/config.py") + assert_path_scope(first, "app/core/config.py", "app/core") assert "app/core/config.py" in second.retrieval_spec.filters.path_scope - assert second.query_plan.sub_intent in {"EXPLAIN_LOCAL", "NEXT_STEPS"} + assert second.query_plan.sub_intent == "GENERIC_QA" layer_ids = [item.layer_id for item in second.retrieval_spec.layer_queries] assert "C3_ENTRYPOINTS" not in layer_ids assert second.evidence_policy.require_flow is False diff --git a/tests/unit_tests/rag/test_intent_router_invariants.py b/tests/unit_tests/rag/test_intent_router_invariants.py index d4ff287..21db7e1 100644 --- a/tests/unit_tests/rag/test_intent_router_invariants.py +++ b/tests/unit_tests/rag/test_intent_router_invariants.py @@ -1,204 +1,121 @@ import pytest -from tests.unit_tests.rag.asserts_intent_router import ( - assert_domain_layer_prefixes, - assert_domains, - assert_file_only_scope, - assert_has_file_path, - assert_intent, - assert_no_symbol_keyword, - assert_no_symbol_leakage_from_paths, - assert_spans_valid, - assert_sub_intent, - assert_test_policy, -) from tests.unit_tests.rag.intent_router_testkit import run_sequence pytestmark = pytest.mark.intent_router -def test_invariant_code_file_path_with_canonical_key_term() -> None: - result = run_sequence(["Уточни по файлу app/core/config.py"])[0] - - assert_intent(result, "CODE_QA") - assert_has_file_path(result, "app/core/config.py") - assert_file_only_scope(result, "app/core/config.py") - key_terms = [anchor.value for anchor in result.query_plan.anchors if anchor.type == "KEY_TERM"] - assert "файл" in key_terms - assert "файлу" not in key_terms - assert_spans_valid(result) - assert_domain_layer_prefixes(result) +def _docs_result(query: str): + result = run_sequence([query])[0] + assert result.docs_routing is not None + assert result.retrieval_plan is not None + return result -def test_invariant_open_file_for_specified_file_phrase_uses_narrow_layers() -> None: - result = run_sequence(["Уточни по файлу app/core/config.py"])[0] +@pytest.mark.parametrize( + ("query", "endpoint"), + [ + ("как работает метод health", "/health"), + ("объясни /health", "/health"), + ("что делает endpoint /send", "/send"), + ], +) +def test_docs_mvp_api_method_explain_cases(query: str, endpoint: str) -> None: + result = _docs_result(query) - assert_intent(result, "CODE_QA") - assert_sub_intent(result, "OPEN_FILE") - assert_file_only_scope(result, "app/core/config.py") - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert layer_ids == ["C0_SOURCE_CHUNKS"] - assert result.evidence_policy.require_flow is False + assert result.docs_routing.sub_intent == "API_METHOD_EXPLAIN" + assert result.docs_routing.intent == "DOCS_QA" + assert result.docs_routing.scope.level == "method" + assert result.docs_routing.anchors.endpoint_path == endpoint + assert result.retrieval_plan.plan_id == "docs_api_method_explain_v1" + assert result.retrieval_plan.filters["endpoint_path"] == endpoint -def test_invariant_inline_code_span_routes_to_code_and_extracts_symbol() -> None: - result = run_sequence(["Уточни по коду `def build(x): return x`"])[0] +@pytest.mark.parametrize( + ("query", "scope_level", "domain_name"), + [ + ("какие есть методы в проекте", "project", None), + ("покажи все api", "project", None), + ("какие методы в notifications", "domain", "notifications"), + ], +) +def test_docs_mvp_list_api_methods_cases(query: str, scope_level: str, domain_name: str | None) -> None: + result = _docs_result(query) - assert_intent(result, "CODE_QA") - assert_spans_valid(result) - assert_no_symbol_keyword(result) - symbols = [anchor.value for anchor in result.query_plan.anchors if anchor.type == "SYMBOL"] - key_terms = [anchor.value for anchor in result.query_plan.anchors if anchor.type == "KEY_TERM"] - assert "build" in symbols - assert "def" in key_terms + assert result.docs_routing.sub_intent == "LIST_API_METHODS" + assert result.docs_routing.intent == "DOCS_DISCOVERY" + assert result.docs_routing.scope.level == scope_level + assert result.retrieval_plan.plan_id == "docs_list_api_methods_v1" + assert result.retrieval_plan.primary_doc_types == ["api_method"] + if domain_name: + assert result.retrieval_plan.filters["domain_name"] == domain_name -def test_invariant_docs_cyrillic_path_with_quotes() -> None: - result = run_sequence(["Что сказано в «docs/архитектура.md»?"])[0] +@pytest.mark.parametrize( + ("query", "domain_name", "subdomain_name", "entity_name"), + [ + ("какие документы есть по notifications", "notifications", None, None), + ("найди документацию по telegram_delivery", "telegram_delivery", None, None), + ("какие документы связаны с health", None, None, "health"), + ], +) +def test_docs_mvp_find_documents_cases( + query: str, + domain_name: str | None, + subdomain_name: str | None, + entity_name: str | None, +) -> None: + result = _docs_result(query) - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "COMPONENT_EXPLAIN") - assert_domains(result, ["DOCS"]) - assert "docs/архитектура.md" in result.query_plan.normalized - assert_has_file_path(result, "docs/архитектура.md") - assert any(anchor.type == "DOC_REF" for anchor in result.query_plan.anchors) - assert result.retrieval_spec.filters.doc_kinds == [] - assert_spans_valid(result) - assert_domain_layer_prefixes(result) + assert result.docs_routing.sub_intent == "FIND_DOCUMENTS_BY_DOMAIN" + assert result.docs_routing.intent == "DOCS_DISCOVERY" + assert result.retrieval_plan.plan_id == "docs_find_documents_by_domain_v1" + if domain_name: + assert result.retrieval_plan.filters["domain_name"] == domain_name + if subdomain_name: + assert result.retrieval_plan.filters["subdomain_name"] == subdomain_name + if entity_name: + assert result.retrieval_plan.filters["entity_name"] == entity_name -def test_invariant_file_check_phrase_not_project_misc() -> None: - result = run_sequence(["Проверь app/modules/rag/explain/intent_builder.py и объясни"])[0] +@pytest.mark.parametrize( + ("query", "scope_level", "endpoint"), + [ + ("сгенерируй openapi по /health", "method", "/health"), + ("собери swagger по notifications", "domain", None), + ("сделай спецификацию api по всему проекту", "project", None), + ], +) +def test_docs_mvp_generate_openapi_cases(query: str, scope_level: str, endpoint: str | None) -> None: + result = _docs_result(query) - assert_intent(result, "CODE_QA") - assert_domains(result, ["CODE"]) - assert_no_symbol_leakage_from_paths(result) - assert_domain_layer_prefixes(result) + assert result.docs_routing.sub_intent == "GENERATE_OPENAPI" + assert result.docs_routing.intent == "DOCS_GENERATION" + assert result.docs_routing.scope.level == scope_level + assert result.retrieval_plan.plan_id == "docs_generate_openapi_v1" + if endpoint: + assert result.retrieval_plan.filters["endpoint_path"] == endpoint -def test_invariant_tests_include_routing() -> None: - result = run_sequence(["Где тесты на ConfigManager?"])[0] +@pytest.mark.parametrize( + "query", + [ + "что делает это приложение", + "как устроен сервис", + "как связаны worker и api", + ], +) +def test_docs_mvp_general_docs_qa_cases(query: str) -> None: + result = _docs_result(query) - assert_intent(result, "CODE_QA") - assert_test_policy(result, "INCLUDE") - symbols = [anchor.value for anchor in result.query_plan.anchors if anchor.type == "SYMBOL"] - key_terms = [anchor.value for anchor in result.query_plan.anchors if anchor.type == "KEY_TERM"] - assert "ConfigManager" in symbols - assert "тест" in key_terms + assert result.docs_routing.sub_intent == "GENERAL_DOCS_QA" + assert result.docs_routing.intent == "DOCS_FALLBACK" + assert result.retrieval_plan.plan_id == "docs_general_docs_qa_v1" -def test_invariant_keyword_hints_and_expansions_for_function_identifier() -> None: - result = run_sequence(["Теперь объясни функцию load_config"])[0] +def test_docs_mvp_retrieval_filters_are_merged_into_legacy_spec() -> None: + result = _docs_result("какие методы в notifications") - assert_intent(result, "CODE_QA") - assert "load_config" in result.query_plan.keyword_hints - assert "функция" not in result.query_plan.keyword_hints - assert "def" not in result.query_plan.expansions - - -def test_invariant_open_file_sub_intent_uses_narrow_retrieval_profile() -> None: - result = run_sequence(["Открой файл app/core/config.py"])[0] - - assert_intent(result, "CODE_QA") - assert_sub_intent(result, "OPEN_FILE") - assert_file_only_scope(result, "app/core/config.py") - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert "C0_SOURCE_CHUNKS" in layer_ids - assert "C1_SYMBOL_CATALOG" not in layer_ids - assert "C2_DEPENDENCY_GRAPH" not in layer_ids - assert "C3_ENTRYPOINTS" not in layer_ids - assert result.evidence_policy.require_flow is False - - -def test_invariant_docs_question_routes_to_docs() -> None: - result = run_sequence(["Что сказано в документации?"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_domains(result, ["DOCS"]) - assert_domain_layer_prefixes(result) - assert result.query_plan.keyword_hints - assert any(item in result.query_plan.expansions for item in result.query_plan.keyword_hints) - - -def test_invariant_docs_flow_sub_intent_uses_workflow_layers() -> None: - result = run_sequence(["Как работает процесс создания заказа по документации?"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "SYSTEM_FLOW_EXPLAIN") - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert layer_ids == ["D4_WORKFLOW_INDEX", "D5_RELATION_GRAPH", "D1_DOCUMENT_CATALOG", "D0_DOC_CHUNKS"] - - -def test_invariant_docs_entity_sub_intent_uses_entity_layers() -> None: - result = run_sequence(["Что такое сущность Order в документации?"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "ENTITY_EXPLAIN") - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert layer_ids == ["D3_ENTITY_CATALOG", "D5_RELATION_GRAPH", "D1_DOCUMENT_CATALOG", "D0_DOC_CHUNKS"] - - -def test_invariant_entity_like_camel_case_prefers_entity_explain() -> None: - result = run_sequence(["Что такое WorkerHealth?"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "ENTITY_EXPLAIN") - - -def test_invariant_related_docs_routes_to_docs_explain() -> None: - result = run_sequence(["Найди документацию по billing"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "RELATED_DOCS_EXPLAIN") - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert layer_ids == ["D5_RELATION_GRAPH", "D1_DOCUMENT_CATALOG", "D0_DOC_CHUNKS"] - - -def test_invariant_docs_navigation_uses_related_docs_explain() -> None: - result = run_sequence(["Что связано с checkout документацией?"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "RELATED_DOCS_EXPLAIN") - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert layer_ids == ["D5_RELATION_GRAPH", "D1_DOCUMENT_CATALOG", "D0_DOC_CHUNKS"] - - -def test_invariant_openapi_routes_to_docs_layers_with_api_filter() -> None: - result = run_sequence(["Сгенерируй openapi yaml для создания заказа"])[0] - - assert_intent(result, "OPENAPI_GENERATION") - assert_sub_intent(result, "OPENAPI_METHOD_GENERATE") - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert layer_ids == ["D1_DOCUMENT_CATALOG", "D2_FACT_INDEX", "D0_DOC_CHUNKS"] - assert result.retrieval_spec.filters.doc_type == "api_method" - - -def test_invariant_general_qa_routes_to_generic_docs_layers() -> None: - result = run_sequence(["Помоги разобраться"])[0] - - assert_intent(result, "GENERAL_QA") - assert_sub_intent(result, "GENERIC_QA") - assert_domains(result, ["DOCS"]) - layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries] - assert layer_ids == ["D1_DOCUMENT_CATALOG", "D0_DOC_CHUNKS"] - - -def test_invariant_component_like_manager_routes_to_component_explain() -> None: - result = run_sequence(["Какую роль в системе играет RuntimeManager?"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "COMPONENT_EXPLAIN") - - -def test_invariant_cycle_query_routes_to_system_flow_explain() -> None: - result = run_sequence(["Объясни как работает цикл отправки уведомлений"])[0] - - assert_intent(result, "DOCUMENTATION_EXPLAIN") - assert_sub_intent(result, "SYSTEM_FLOW_EXPLAIN") - - -def test_invariant_overview_question_routes_to_general_qa() -> None: - result = run_sequence(["Что вообще описано в документации по этому сервису?"])[0] - - assert_intent(result, "GENERAL_QA") - assert_sub_intent(result, "GENERIC_QA") + assert getattr(result.retrieval_spec.filters, "doc_type", None) == "api_method" + assert getattr(result.retrieval_spec.filters, "domain_name", None) == "notifications" + assert getattr(result.retrieval_spec.filters, "scope_level", None) == "domain" diff --git a/tests/unit_tests/rag/test_intent_router_llm_disambiguation.py b/tests/unit_tests/rag/test_intent_router_llm_disambiguation.py index d3b9dfa..040c483 100644 --- a/tests/unit_tests/rag/test_intent_router_llm_disambiguation.py +++ b/tests/unit_tests/rag/test_intent_router_llm_disambiguation.py @@ -2,9 +2,9 @@ from __future__ import annotations import json -from app.modules.agent.intent_router_v2 import ConversationState, IntentRouterV2 -from app.modules.agent.intent_router_v2.intent.classifier import IntentClassifierV2 -from app.modules.agent.intent_router_v2.intent.llm_disambiguator import DocsLlmDisambiguator +from app.core.agent.intent_router import ConversationState, IntentRouterV2 +from app.core.agent.intent_router.docs_mvp.llm_classifier import DocsMvpLlmClassifier +from app.core.agent.intent_router.intent.classifier import IntentClassifierV2 from tests.unit_tests.rag.intent_router_testkit import repo_context @@ -21,75 +21,62 @@ class FakeLlm: return self.response -def test_technical_query_keeps_deterministic_routing_without_llm_call() -> None: - llm = FakeLlm('{"sub_intent":"GENERIC_QA","reason":"unused","confidence":"low"}') - router = IntentRouterV2( +def _router(llm: FakeLlm) -> IntentRouterV2: + return IntentRouterV2( classifier=IntentClassifierV2(), - llm_disambiguator=DocsLlmDisambiguator(llm), + docs_llm_classifier=DocsMvpLlmClassifier(llm), enable_llm_disambiguation=True, ) - result = router.route("Объясни endpoint /health", ConversationState(), repo_context()) - assert result.query_plan.sub_intent == "API_METHOD_EXPLAIN" - assert result.is_ambiguous is False - assert result.routing_mode == "deterministic" +def test_docs_technical_query_keeps_deterministic_routing_without_llm_call() -> None: + llm = FakeLlm("{}") + result = _router(llm).route("Объясни endpoint /health", ConversationState(), repo_context()) + + assert result.docs_routing is not None + assert result.docs_routing.sub_intent == "API_METHOD_EXPLAIN" + assert result.docs_routing.routing_mode == "deterministic" assert result.llm_router_used is False assert llm.calls == [] def test_ambiguous_query_can_be_resolved_by_llm() -> None: - llm = FakeLlm('{"sub_intent":"ENTITY_EXPLAIN","reason":"runtime health is a concept/entity here","confidence":"medium"}') - router = IntentRouterV2( - classifier=IntentClassifierV2(), - llm_disambiguator=DocsLlmDisambiguator(llm), - enable_llm_disambiguation=True, + llm = FakeLlm( + json.dumps( + { + "intent": "DOCS_DISCOVERY", + "sub_intent": "FIND_DOCUMENTS_BY_DOMAIN", + "confidence": 0.83, + "anchors": {"entity_name": "health", "doc_query": "документация по health"}, + "scope": {"level": "domain"}, + "reason_short": "health here is a docs topic", + }, + ensure_ascii=False, + ) ) + result = _router(llm).route("документация по health", ConversationState(), repo_context()) - result = router.route("Объясни runtime health", ConversationState(), repo_context()) - - assert result.is_ambiguous is True - assert result.routing_mode == "llm_disambiguation" + assert result.docs_routing is not None + assert result.docs_routing.routing_mode == "llm_assisted" + assert result.docs_routing.sub_intent == "FIND_DOCUMENTS_BY_DOMAIN" + assert result.retrieval_plan is not None + assert result.retrieval_plan.plan_id == "docs_find_documents_by_domain_v1" assert result.llm_router_used is True - assert result.deterministic_selected_sub_intent - assert result.llm_router_selected_sub_intent == "ENTITY_EXPLAIN" - assert result.query_plan.sub_intent == "ENTITY_EXPLAIN" -def test_ambiguous_query_falls_back_to_deterministic_when_llm_fails() -> None: +def test_ambiguous_query_falls_back_to_general_docs_when_llm_fails() -> None: llm = FakeLlm("{}", fail=True) - router = IntentRouterV2( - classifier=IntentClassifierV2(), - llm_disambiguator=DocsLlmDisambiguator(llm), - enable_llm_disambiguation=True, - ) + result = _router(llm).route("health документация", ConversationState(), repo_context()) - result = router.route("Как работает health check runtime?", ConversationState(), repo_context()) - - assert result.is_ambiguous is True - assert result.routing_mode == "deterministic_fallback" - assert result.llm_router_used is False - assert result.llm_router_error == "llm unavailable" - assert result.query_plan.sub_intent == result.deterministic_selected_sub_intent + assert result.docs_routing is not None + assert result.docs_routing.routing_mode == "llm_fallback" + assert result.docs_routing.sub_intent == "GENERAL_DOCS_QA" + assert result.retrieval_plan is not None + assert result.retrieval_plan.plan_id == "docs_general_docs_qa_v1" -def test_overview_query_stays_in_generic_qa() -> None: - llm = FakeLlm('{"sub_intent":"GENERIC_QA","reason":"overview query","confidence":"high"}') - router = IntentRouterV2( - classifier=IntentClassifierV2(), - llm_disambiguator=DocsLlmDisambiguator(llm), - enable_llm_disambiguation=True, - ) +def test_llm_classifier_rejects_unknown_labels() -> None: + llm = FakeLlm(json.dumps({"intent": "DOCS_QA", "sub_intent": "MADE_UP"})) + classifier = DocsMvpLlmClassifier(llm) - result = router.route("Какая структура документации?", ConversationState(), repo_context()) - - assert result.is_ambiguous is False or result.query_plan.sub_intent == "GENERIC_QA" - assert result.intent == "GENERAL_QA" - assert result.query_plan.sub_intent == "GENERIC_QA" - - -def test_llm_disambiguator_rejects_unknown_labels() -> None: - llm = FakeLlm(json.dumps({"sub_intent": "MADE_UP", "reason": "bad", "confidence": "high"})) - disambiguator = DocsLlmDisambiguator(llm) - - assert disambiguator.choose({"query": "test"}) is None + assert classifier.classify({"query": "test"}) is None diff --git a/tests/unit_tests/rag/test_layered_gateway.py b/tests/unit_tests/rag/test_layered_gateway.py index 244d39b..43982ee 100644 --- a/tests/unit_tests/rag/test_layered_gateway.py +++ b/tests/unit_tests/rag/test_layered_gateway.py @@ -1,4 +1,4 @@ -from app.modules.agent.runtime.steps.explain.layered_gateway import LayeredRetrievalGateway +from app.core.agent.runtime.steps.explain.layered_gateway import LayeredRetrievalGateway class _Embedder: diff --git a/tests/unit_tests/rag/test_path_filter.py b/tests/unit_tests/rag/test_path_filter.py index 7472496..e97f1dc 100644 --- a/tests/unit_tests/rag/test_path_filter.py +++ b/tests/unit_tests/rag/test_path_filter.py @@ -1,6 +1,6 @@ from __future__ import annotations -from app.modules.rag.indexing.common.path_filter import ( +from app.core.rag.indexing.common.path_filter import ( count_indexable_change_upserts, filter_changes_for_indexing, filter_snapshot_files, diff --git a/tests/unit_tests/rag/test_query_normalization.py b/tests/unit_tests/rag/test_query_normalization.py index 251509a..9bf52ce 100644 --- a/tests/unit_tests/rag/test_query_normalization.py +++ b/tests/unit_tests/rag/test_query_normalization.py @@ -1,6 +1,6 @@ import pytest -from app.modules.agent.intent_router_v2.analysis.normalization import QueryNormalizer +from app.core.agent.intent_router.analysis.normalization import QueryNormalizer pytestmark = pytest.mark.intent_router diff --git a/tests/unit_tests/rag/test_query_terms.py b/tests/unit_tests/rag/test_query_terms.py index 654ec3c..19494ff 100644 --- a/tests/unit_tests/rag/test_query_terms.py +++ b/tests/unit_tests/rag/test_query_terms.py @@ -1,4 +1,4 @@ -from app.modules.rag.retrieval.query_terms import extract_query_terms +from app.core.rag.retrieval.query_terms import extract_query_terms def test_extract_query_terms_from_code_question() -> None: diff --git a/tests/unit_tests/rag/test_rag_service_filtering.py b/tests/unit_tests/rag/test_rag_service_filtering.py index 3897e98..8dd28ad 100644 --- a/tests/unit_tests/rag/test_rag_service_filtering.py +++ b/tests/unit_tests/rag/test_rag_service_filtering.py @@ -2,7 +2,8 @@ from __future__ import annotations import asyncio -from app.modules.rag.services.rag_service import RagService +from app.core.rag.contracts.enums import RagLayer +from app.core.rag.indexing.service import RagService class _FakeEmbedder: @@ -50,3 +51,40 @@ def test_rag_service_progress_uses_only_indexable_files() -> None: assert cache_hits == 0 assert cache_misses == 1 assert progress == [(1, 1, "src/main.py")] + + +def test_rag_service_keeps_docs_artifact_type_metadata() -> None: + repository = _FakeRepository() + service = RagService(embedder=_FakeEmbedder(), repository=repository) + files = [ + { + "path": "docs/api/health.md", + "content_hash": "docs-h1", + "content": """--- +id: api.health +type: api_method +doc_type: api_method +title: Health API +domain: runtime +sub_domain: health +related_docs: [] +status: active +--- +# Health API + +## Summary +- Purpose: check service health. + +## Details +### Описание +Returns health payload. +""", + } + ] + + asyncio.run(service.index_snapshot("project-1", files)) + + doc_chunk = next(doc for doc in repository.replaced_docs if doc.layer == RagLayer.DOCS_DOC_CHUNKS) + assert doc_chunk.metadata["artifact_type"] == "DOCS" + assert doc_chunk.metadata["domain"] == "runtime" + assert doc_chunk.metadata["subdomain"] == "health" diff --git a/tests/unit_tests/rag/test_rag_session_retriever.py b/tests/unit_tests/rag/test_rag_session_retriever.py new file mode 100644 index 0000000..5902aac --- /dev/null +++ b/tests/unit_tests/rag/test_rag_session_retriever.py @@ -0,0 +1,26 @@ +"""Smoke-тест стандартного retrieval API: один embed и вызов repository.""" + +from __future__ import annotations + +import asyncio +from unittest.mock import MagicMock + +from app.core.rag.embedding.gigachat_embedder import GigaChatEmbedder +from app.core.rag.retrieval.session_retriever import RagSessionRetriever, RetrievalPlan + + +def test_rag_session_retriever_calls_repository() -> None: + embedder = MagicMock(spec=GigaChatEmbedder) + embedder.embed = MagicMock(return_value=[[0.1, 0.2]]) + repo = MagicMock() + repo.retrieve = MagicMock(return_value=[{"path": "a.md", "layer": "D0_DOC_CHUNKS"}]) + retriever = RagSessionRetriever(repository=repo, embedder=embedder) + plan = RetrievalPlan(profile="test", layers=["D0_DOC_CHUNKS", "D1_DOCUMENT_CATALOG"], limit=5) + rows = asyncio.run(retriever.retrieve("sid-1", "hello", plan)) + assert len(rows) == 1 + assert embedder.embed.called + assert repo.retrieve.called + call_kw = repo.retrieve.call_args + assert call_kw[0][0] == "sid-1" + assert call_kw[1]["layers"] == plan.layers + assert call_kw[1]["limit"] == 5 diff --git a/tests/unit_tests/rag/test_retrieval_statement_builder.py b/tests/unit_tests/rag/test_retrieval_statement_builder.py index b0c8dce..7bbb6c6 100644 --- a/tests/unit_tests/rag/test_retrieval_statement_builder.py +++ b/tests/unit_tests/rag/test_retrieval_statement_builder.py @@ -1,5 +1,5 @@ -from app.modules.rag.persistence.retrieval_statement_builder import RetrievalStatementBuilder -from app.modules.rag.retrieval.test_filter import build_test_filters, is_test_path +from app.core.rag.persistence.retrieval_statement_builder import RetrievalStatementBuilder +from app.core.rag.retrieval.test_filter import build_test_filters, is_test_path def test_retrieve_builder_adds_test_exclusion_filters() -> None: diff --git a/tests/unit_tests/rag/test_retriever_v2_no_fallback.py b/tests/unit_tests/rag/test_retriever_v2_no_fallback.py index 14ffa2e..1d91f81 100644 --- a/tests/unit_tests/rag/test_retriever_v2_no_fallback.py +++ b/tests/unit_tests/rag/test_retriever_v2_no_fallback.py @@ -1,4 +1,4 @@ -from app.modules.agent.runtime.steps.explain import CodeExplainRetrieverV2, LayeredRetrievalGateway +from app.core.agent.runtime.steps.explain import CodeExplainRetrieverV2, LayeredRetrievalGateway class _ExplodingEmbedder: diff --git a/tests/unit_tests/rag/test_retriever_v2_pack.py b/tests/unit_tests/rag/test_retriever_v2_pack.py index 2b7ee28..c93aacf 100644 --- a/tests/unit_tests/rag/test_retriever_v2_pack.py +++ b/tests/unit_tests/rag/test_retriever_v2_pack.py @@ -1,5 +1,5 @@ -from app.modules.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem -from app.modules.agent.runtime.steps.explain.retriever_v2 import CodeExplainRetrieverV2 +from app.core.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem +from app.core.agent.runtime.steps.explain.retriever_v2 import CodeExplainRetrieverV2 class _FakeGateway: diff --git a/tests/unit_tests/rag/test_retriever_v2_production_first.py b/tests/unit_tests/rag/test_retriever_v2_production_first.py index 151abab..8ba37a3 100644 --- a/tests/unit_tests/rag/test_retriever_v2_production_first.py +++ b/tests/unit_tests/rag/test_retriever_v2_production_first.py @@ -1,7 +1,7 @@ from types import SimpleNamespace -from app.modules.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem -from app.modules.agent.runtime.steps.explain.retriever_v2 import CodeExplainRetrieverV2 +from app.core.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem +from app.core.agent.runtime.steps.explain.retriever_v2 import CodeExplainRetrieverV2 class _ProductionFirstGateway: diff --git a/tests/unit_tests/rag/test_trace_builder.py b/tests/unit_tests/rag/test_trace_builder.py index 2527311..47581c9 100644 --- a/tests/unit_tests/rag/test_trace_builder.py +++ b/tests/unit_tests/rag/test_trace_builder.py @@ -1,5 +1,5 @@ -from app.modules.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem -from app.modules.agent.runtime.steps.explain.trace_builder import TraceBuilder +from app.core.agent.runtime.steps.explain.models import CodeLocation, LayeredRetrievalItem +from app.core.agent.runtime.steps.explain.trace_builder import TraceBuilder class _FakeGraphRepository: