agent/tests/pipeline_setup_v4/README.md

pipeline_setup_v4

pipeline_setup_v4 is a component-level test harness built on top of the v3 ideas.

Differences from v3:

• each YAML case targets a single isolated component;
• results are written next to the suite in cases/.../test_runs/...;
• supported components are process_v2_intent_router and process_v2_retrieval_policy_resolver. Also available: process_v2_router_plus_retrieval_policy for the linked route -> plan chain, process_v2_router_plus_retrieval_policy_rag for the linked route -> plan -> rag chain, and process_v2_full_chain for the full route -> plan -> rag -> evidence -> workflow LLM chain.

Run

PYTHONPATH=. python -m tests.pipeline_setup_v4.run \
  --cases-dir tests/pipeline_setup_v4/cases/suite_01/process_v2_intent_router/router_only_docs_v2_matrix.yaml \
  --run-name smoke_v4

LLM-first router suite:

PYTHONPATH=. python -m tests.pipeline_setup_v4.run \
  --cases-dir tests/pipeline_setup_v4/cases/suite_02/process_v2_intent_router/router_llm_first_v3.yaml \
  --run-name llm_first_v3

Retrieval policy resolver suite:

PYTHONPATH=. python -m tests.pipeline_setup_v4.run \
  --cases-dir tests/pipeline_setup_v4/cases/suite_03/process_v2_retrieval_policy_resolver/cases.yaml \
  --run-name retrieval_policy_v1

Linked router + retrieval policy suite:

PYTHONPATH=. python3 -m tests.pipeline_setup_v4.run \
  --cases-dir tests/pipeline_setup_v4/cases/suite_04/process_v2_router_plus_retrieval_policy \
  --run-name router_plus_policy_v1

Inside suite_04, cases are split into:

• strict_regression_cases.yaml for contract-level invariants
• soft_observational_cases.yaml for LLM-sensitive boundary scenarios

Quality-gate mini-pack:

PYTHONPATH=. python3 -m tests.pipeline_setup_v4.run \
  --cases-dir tests/pipeline_setup_v4/cases/suite_05/process_v2_router_plus_retrieval_policy_quality_gate/cases.yaml \
  --run-name router_plus_policy_qg_v1

Linked router + retrieval policy + rag suite:

PYTHONPATH=src:. DATABASE_URL='postgresql+psycopg://agent:agent@127.0.0.1:5432/agent' python3 -m tests.pipeline_setup_v4.run \
  --cases-dir tests/pipeline_setup_v4/cases/suite_06/process_v2_router_plus_retrieval_policy_rag/cases.yaml \
  --run-name router_plus_policy_rag_v1

Full process v2 chain with workflow LLM:

PYTHONPATH=src:. DATABASE_URL='postgresql+psycopg://agent:agent@127.0.0.1:5432/agent' python3 -m tests.pipeline_setup_v4.run \
  --cases-dir tests/pipeline_setup_v4/cases/suite_07/process_v2_full_chain/cases.yaml \
  --run-name process_v2_full_chain_v1