PLBA2-2. Реализация
This commit is contained in:
@@ -1,35 +1,94 @@
|
||||
# PLBA v2
|
||||
|
||||
PLBA v2 is a compact, transport-neutral runtime for business microservices. It owns application lifecycle, worker supervision, versioned configuration snapshots, traces, structured logging, health aggregation, workflows and an optional audit port.
|
||||
PLBA v2 is a transport-neutral Python library for application lifecycle,
|
||||
workers, health aggregation, configuration snapshots, workflows, business
|
||||
tracing, and structured technical logs. It is a separate package from PLBA v1.
|
||||
|
||||
It deliberately does not own task queues, RabbitMQ clients, gRPC servers, HTTP servers, a database driver, or business logic. A service supplies those as adapters and owns its entrypoint.
|
||||
The application owns its entrypoint and composition root, business logic, HTTP
|
||||
host and listener, database schema and migrations, and message queues. PLBA2
|
||||
provides components and adapters that the application wires together; it does
|
||||
not start a service or create a second HTTP port.
|
||||
|
||||
## Status
|
||||
## Capabilities
|
||||
|
||||
This repository contains the first executable skeleton and public contracts. It is a breaking, separate package from PLBA v1; no v1 compatibility facade is provided.
|
||||
- `runtime/application-identity` resolves `application_id` from `config.yaml`,
|
||||
then `[project].name` in `pyproject.toml`, then UUIDv4.
|
||||
- `runtime/worker` registers uniquely named workers, applies the `disabled`
|
||||
flag, runs sequential scheduled cycles, combines application and automatic
|
||||
health, and supports graceful or force stop.
|
||||
- `runtime/application-lifecycle` starts and stops all enabled workers and
|
||||
reports application lifecycle states.
|
||||
- `health/aggregation` combines worker and contributor states, criticality,
|
||||
missing components, and application lifecycle into a health report.
|
||||
- `api/http/control-api` is a mountable ASGI adapter for `GET /health`,
|
||||
`POST /runtime/start`, `POST /runtime/stop`, and `GET /runtime/status`.
|
||||
- `configuration/application-config-file` loads an application-typed root
|
||||
`config.yaml` and atomically reloads valid snapshots without a restart.
|
||||
- `configuration/application-settings` provides a typed `SettingsReader` and a
|
||||
PostgreSQL adapter for the application's `settings` table.
|
||||
- `workflow/orchestration` executes named object-based steps through declared
|
||||
transitions and persists run and step snapshots through an application port.
|
||||
- `observability/business-tracing` provides trace contracts and a PostgreSQL
|
||||
writer for application-owned `trace_contexts` and `trace_messages` tables.
|
||||
- `observability/technical-logging` formats JSON log records with application
|
||||
and business trace identifiers for stdout.
|
||||
|
||||
## Run checks
|
||||
## Install
|
||||
|
||||
```bash
|
||||
python -m unittest discover -s tests -v
|
||||
pip install .
|
||||
```
|
||||
|
||||
## Configuration-service shape
|
||||
The PostgreSQL adapters accept the application's pool and schema. PLBA2 does
|
||||
not install or manage a database driver. The HTTP adapter uses ASGI and does
|
||||
not install or manage an HTTP server.
|
||||
|
||||
## Composition outline
|
||||
|
||||
The application loads and validates configuration before building dependent
|
||||
components. It supplies a decoder for its own typed fields:
|
||||
|
||||
```python
|
||||
runtime = Runtime(
|
||||
configuration=ConfigurationSnapshotManager(
|
||||
client=configuration_grpc_adapter,
|
||||
scope="configuration-service",
|
||||
validator=validate_configuration,
|
||||
),
|
||||
workers=[grpc_server_worker, configuration_changed_consumer],
|
||||
config_loader = ApplicationConfigLoader(
|
||||
Path("config.yaml"), application_decoder=ServiceConfig.from_mapping
|
||||
)
|
||||
config_manager = ApplicationConfigManager(config_loader)
|
||||
initial_config = config_manager.load_initial()
|
||||
|
||||
await runtime.start()
|
||||
# The RabbitMQ adapter invokes the registered callback after configuration.changed.
|
||||
# The manager refetches and atomically swaps the validated snapshot.
|
||||
await runtime.stop(timeout_seconds=20)
|
||||
identity = ApplicationIdentityResolver(
|
||||
TomlProjectNameReader(Path("pyproject.toml"))
|
||||
)
|
||||
runtime_context = identity.resolve(initial_config.application_id)
|
||||
|
||||
workers = WorkerRegistry()
|
||||
workers.register(WorkerDefinition("inventory_sync", sync_inventory))
|
||||
supervisor = WorkerSupervisor(workers, config_manager)
|
||||
runtime = ApplicationRuntime(runtime_context, supervisor)
|
||||
```
|
||||
|
||||
The adapter authenticates using bootstrap values read by the service from environment/secrets. Business code reads `manager.current()` and never makes a synchronous configuration-service call.
|
||||
The application creates its HTTP host and mounts `ControlApi` on the same
|
||||
listener. `RuntimeHealthInputProvider` connects the runtime, worker supervisor,
|
||||
and registered health contributors to `HealthAggregator`. The application
|
||||
starts its own server and chooses when to call `runtime.start()` and
|
||||
`runtime.stop()`.
|
||||
|
||||
For configuration changes, run `config_manager.watch(...)` in an application
|
||||
task and handle reload errors in the supplied callback. Existing snapshots
|
||||
remain stable; an invalid update leaves the latest valid snapshot active.
|
||||
|
||||
## Database ownership
|
||||
|
||||
The application creates and seeds `settings` in its migrations. It also creates
|
||||
`trace_contexts` and `trace_messages` in its migrations. PLBA2 does not create
|
||||
tables, run application migrations, copy YAML into the database, expose a
|
||||
business-history API, publish business events to RabbitMQ, or provide shared
|
||||
worker-message queues. See [database contracts](docs/database-contracts.md) and
|
||||
[configuration](docs/configuration.md) for the consumer-facing contracts.
|
||||
See [integration guidance](docs/integration.md) for wiring the HTTP, database,
|
||||
workflow, trace, and logging adapters into an application.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
python3 -m pytest
|
||||
```
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Application configuration
|
||||
|
||||
PLBA2 reads `config.yaml` from the application root. The application supplies a
|
||||
typed decoder for its own fields; PLBA2 validates the common identity and worker
|
||||
fields and keeps the application model in the same snapshot.
|
||||
|
||||
```yaml
|
||||
application_id: inventory-service
|
||||
workers:
|
||||
inventory_sync:
|
||||
disabled: false
|
||||
interval_seconds: 60
|
||||
cleanup:
|
||||
pause_seconds: 300
|
||||
database_url: "..."
|
||||
```
|
||||
|
||||
`application_id` is optional. A non-empty value is used unchanged; otherwise
|
||||
identity resolution uses `[project].name` from `pyproject.toml`, then a UUIDv4.
|
||||
The resolved value is passed to `ApplicationRuntime` through `RuntimeContext`
|
||||
before dependent components start.
|
||||
|
||||
Each worker entry may set `disabled` (default `false`) and one schedule:
|
||||
`interval_seconds` or `pause_seconds`. Enabled workers need exactly one positive
|
||||
finite schedule. A disabled worker is skipped at application start and does not
|
||||
need a schedule. Changing `disabled` takes effect on the next application start;
|
||||
valid schedule updates are used when the next worker cycle is planned.
|
||||
|
||||
The application defines every other YAML field and its typed model. This file is
|
||||
for rarely changed technical parameters. Business parameters, feature toggles,
|
||||
and technical values that must change remotely while the process is running
|
||||
belong in the application-owned `settings` table, read through `SettingsReader`.
|
||||
|
||||
Load and validate the initial file before starting dependent components. A
|
||||
missing, malformed, or invalid initial file raises `ApplicationConfigLoadError`
|
||||
and blocks startup. After startup, `ApplicationConfigManager.watch` checks for
|
||||
file changes. It swaps in a fully validated snapshot atomically; previously
|
||||
returned snapshots remain unchanged. If an update is invalid or unreadable,
|
||||
`reload_if_changed` returns a failure and the manager keeps the last valid
|
||||
snapshot. The watcher passes that error to the application's callback.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Database contracts
|
||||
|
||||
PLBA2 reads and writes application-owned tables through adapters. The application
|
||||
owns the schema, migrations, initial values, and retention policy. PLBA2 does not
|
||||
create tables or run migrations.
|
||||
|
||||
## `settings`
|
||||
|
||||
The application creates `settings` in its own schema and loads initial rows in
|
||||
its migration. The table has this contract:
|
||||
|
||||
| Field | Type | Constraints | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `id` | `BIGINT` | Identity primary key, not null | Row identifier and final ordering key |
|
||||
| `config_key` | `VARCHAR(128)` | Nullable | Application-defined scope or entity |
|
||||
| `setting_key` | `VARCHAR(128)` | Not null | Stable setting name |
|
||||
| `setting_value` | `TEXT` | Not null | Text value decoded by the application |
|
||||
| `sort_order` | `INT` | Not null, default `0` | Order for repeated keys |
|
||||
| `is_enabled` | `BOOLEAN` | Not null, default `TRUE` | Includes or excludes a row from reads |
|
||||
| `description` | `TEXT` | Nullable | Human-readable explanation |
|
||||
|
||||
Indexes should support the adapter's filters and ordering, for example:
|
||||
|
||||
```sql
|
||||
CREATE INDEX settings_key_enabled_order_idx
|
||||
ON settings (setting_key, is_enabled, sort_order, id);
|
||||
CREATE INDEX settings_scope_lookup_order_idx
|
||||
ON settings (setting_key, config_key, is_enabled, sort_order, id);
|
||||
```
|
||||
|
||||
There is no uniqueness constraint on `setting_key`; several enabled rows may
|
||||
form an ordered list. Reads filter by `setting_key`, an optional `config_key`,
|
||||
and `is_enabled = TRUE`, then order by `sort_order`, `id`. The value of a feature
|
||||
toggle belongs in `setting_value`; `is_enabled` only controls whether the row is
|
||||
read. `config_key` relationships are owned by the application.
|
||||
|
||||
`settings` stores business parameters, feature toggles, and technical values
|
||||
that must change remotely while the process runs. Rarely changed technical
|
||||
parameters remain in the root `config.yaml`; PLBA2 does not copy YAML into the
|
||||
database.
|
||||
|
||||
## `trace_contexts` and `trace_messages`
|
||||
|
||||
The application creates both trace tables in its own schema through its own
|
||||
migration. Public code calls the context identifier `business_trace_id`; both
|
||||
tables store it as `trace_id`.
|
||||
|
||||
`trace_contexts` contains:
|
||||
|
||||
| Field | Type | Constraints | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `trace_id` | `UUID` | Primary key, not null | Public `business_trace_id` |
|
||||
| `parent_id` | `UUID` | Nullable | Parent context identifier |
|
||||
| `alias` | `TEXT` | Not null | Short context name |
|
||||
| `trace_type` | `TEXT` | Not null | Business operation type |
|
||||
| `event_time` | `TIMESTAMPTZ` | Not null | Context creation time |
|
||||
| `attrs_json` | `JSONB` | Not null | Context attributes, including `application_id` |
|
||||
|
||||
Create indexes on `parent_id` and `event_time`.
|
||||
|
||||
`trace_messages` contains:
|
||||
|
||||
| Field | Type | Constraints | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `id` | `BIGINT` | Identity primary key, not null | Message identifier |
|
||||
| `trace_id` | `UUID` | Not null, references `trace_contexts.trace_id` with `ON DELETE CASCADE` | Owning context |
|
||||
| `event_time` | `TIMESTAMPTZ` | Not null | Message time |
|
||||
| `step` | `TEXT` | Not null | Business step name |
|
||||
| `status` | `TEXT` | Not null | Step result |
|
||||
| `level` | `TEXT` | Not null | Business message level |
|
||||
| `message` | `TEXT` | Not null | Human-readable business message |
|
||||
| `attrs_json` | `JSONB` | Not null | Message attributes, including `application_id` |
|
||||
|
||||
Create indexes on `trace_id`, `event_time`, and `status`. The PostgreSQL
|
||||
adapter only inserts rows, adds the runtime `application_id` to each
|
||||
`attrs_json`, and passes database failures to the application. It does not
|
||||
create tables, run migrations, publish RabbitMQ events, or expose a history API.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Integrating PLBA2
|
||||
|
||||
The application keeps its entrypoint and composes the library before it starts
|
||||
its own HTTP host. The root `config.yaml` must load successfully before
|
||||
dependent components are created.
|
||||
|
||||
## HTTP, runtime, and health
|
||||
|
||||
Register `WorkerDefinition` objects in a `WorkerRegistry`, pass the registry and
|
||||
`ApplicationConfigManager` to `WorkerSupervisor`, resolve an
|
||||
`ApplicationIdentityResolver` result into `RuntimeContext`, and construct
|
||||
`ApplicationRuntime`. Register non-worker checks in `HealthContributorRegistry`
|
||||
and connect it to `HealthAggregator` with `RuntimeHealthInputProvider`.
|
||||
|
||||
Mount `ControlApi(runtime, health_aggregator)` in the application's ASGI host.
|
||||
The host owns the socket and listener; the adapter contributes `/health`,
|
||||
`/runtime/start`, `/runtime/stop`, and `/runtime/status` to that same listener.
|
||||
The application chooses when to start and stop the runtime.
|
||||
|
||||
## Configuration and settings
|
||||
|
||||
Pass an application decoder to `ApplicationConfigLoader`. Its typed model
|
||||
validates application-specific fields while PLBA2 validates `application_id`
|
||||
and worker settings. `ApplicationConfigManager` implements
|
||||
`WorkerConfigReader`; its watcher atomically swaps valid snapshots and passes
|
||||
reload failures to the application's callback.
|
||||
|
||||
Create `settings` and seed its initial values in an application migration. Pass
|
||||
the application's pool and schema to `PostgresSettingsRepository`, then expose
|
||||
it through `SettingsReader` and typed `SettingKey` definitions. The adapter
|
||||
filters enabled rows and returns values in `sort_order`, `id` order. The full
|
||||
schema contract is in [database-contracts.md](database-contracts.md).
|
||||
|
||||
## Workflow and business trace
|
||||
|
||||
Define a `WorkflowDefinition` from named `WorkflowNode` objects and explicit
|
||||
transition maps. Give `WorkflowEngine` an application-owned implementation of
|
||||
`WorkflowPersistence`; that adapter decides how and where to store snapshots.
|
||||
PLBA2 does not connect workflow persistence to a database itself.
|
||||
|
||||
Create `trace_contexts` and `trace_messages` in the application's migrations.
|
||||
Pass the application's pool, schema, and resolved application ID to
|
||||
`PostgresBusinessTraceWriter`. Bind the public trace ID with
|
||||
`business_trace_context(trace_id)` around async business work so technical logs
|
||||
inherit it.
|
||||
|
||||
## Technical logs
|
||||
|
||||
Attach `create_stdout_handler(runtime_context.application_id)` to the
|
||||
application's logger. It uses the standard library `StreamHandler` and emits
|
||||
one JSON object per record. The platform remains responsible for log delivery
|
||||
and retention.
|
||||
+5
-2
@@ -5,10 +5,13 @@ build-backend = "setuptools.build_meta"
|
||||
[project]
|
||||
name = "plba-v2"
|
||||
version = "0.1.0"
|
||||
description = "Portable lifecycle and observability runtime for business microservices"
|
||||
description = "Portable application lifecycle and observability components"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
dependencies = ["PyYAML>=6.0"]
|
||||
|
||||
[project.optional-dependencies]
|
||||
test = ["pytest>=8.0"]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Portable lifecycle and observability components for Python applications."""
|
||||
|
||||
from plba.runtime.context import RuntimeContext
|
||||
|
||||
__all__ = ["RuntimeContext"]
|
||||
Binary file not shown.
@@ -0,0 +1,18 @@
|
||||
from plba.config.loader import ApplicationConfigLoadError, ApplicationConfigLoader
|
||||
from plba.config.manager import (
|
||||
ApplicationConfigManager,
|
||||
ApplicationConfigUnavailable,
|
||||
ConfigReloadResult,
|
||||
ConfigReloadStatus,
|
||||
)
|
||||
from plba.config.models import ApplicationConfigSnapshot
|
||||
|
||||
__all__ = [
|
||||
"ApplicationConfigLoadError",
|
||||
"ApplicationConfigLoader",
|
||||
"ApplicationConfigManager",
|
||||
"ApplicationConfigSnapshot",
|
||||
"ApplicationConfigUnavailable",
|
||||
"ConfigReloadResult",
|
||||
"ConfigReloadStatus",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,106 @@
|
||||
from collections.abc import Callable, Mapping
|
||||
import math
|
||||
from pathlib import Path
|
||||
from typing import Any, Generic, TypeVar
|
||||
|
||||
import yaml
|
||||
|
||||
from plba.config.models import LoadedApplicationConfig
|
||||
from plba.runtime.worker.config import WorkerConfig, WorkerConfigurationError
|
||||
|
||||
T = TypeVar("T")
|
||||
ApplicationConfigDecoder = Callable[[Mapping[str, Any]], T]
|
||||
|
||||
|
||||
class ApplicationConfigLoadError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class ApplicationConfigLoader(Generic[T]):
|
||||
_WORKER_FIELDS = {"disabled", "interval_seconds", "pause_seconds"}
|
||||
|
||||
def __init__(
|
||||
self, config_path: Path, application_decoder: ApplicationConfigDecoder[T]
|
||||
) -> None:
|
||||
self._config_path = config_path
|
||||
self._application_decoder = application_decoder
|
||||
|
||||
@property
|
||||
def config_path(self) -> Path:
|
||||
return self._config_path
|
||||
|
||||
def signature(self) -> tuple[int, int, int, int]:
|
||||
stat = self._config_path.stat()
|
||||
return stat.st_mtime_ns, stat.st_ctime_ns, stat.st_size, stat.st_ino
|
||||
|
||||
def load(self) -> LoadedApplicationConfig[T]:
|
||||
try:
|
||||
with self._config_path.open("r", encoding="utf-8") as config_file:
|
||||
raw_config = yaml.safe_load(config_file)
|
||||
except (OSError, UnicodeDecodeError, yaml.YAMLError) as error:
|
||||
raise ApplicationConfigLoadError("could not read config.yaml") from error
|
||||
if not isinstance(raw_config, dict):
|
||||
raise ApplicationConfigLoadError("config.yaml root must be a mapping")
|
||||
try:
|
||||
application_id = self._application_id(raw_config)
|
||||
workers = self._workers(raw_config)
|
||||
application_config = self._application_decoder(raw_config)
|
||||
except ApplicationConfigLoadError:
|
||||
raise
|
||||
except Exception as error:
|
||||
raise ApplicationConfigLoadError("config.yaml does not match the application model") from error
|
||||
return LoadedApplicationConfig(application_id, workers, application_config)
|
||||
|
||||
@staticmethod
|
||||
def _application_id(raw_config: Mapping[str, Any]) -> str | None:
|
||||
if "application_id" not in raw_config:
|
||||
return None
|
||||
application_id = raw_config["application_id"]
|
||||
if not isinstance(application_id, str):
|
||||
raise ApplicationConfigLoadError("application_id must be a string")
|
||||
return application_id
|
||||
|
||||
@staticmethod
|
||||
def _workers(raw_config: Mapping[str, Any]) -> dict[str, WorkerConfig]:
|
||||
raw_workers = raw_config.get("workers", {})
|
||||
if not isinstance(raw_workers, dict):
|
||||
raise ApplicationConfigLoadError("workers must be a mapping")
|
||||
workers: dict[str, WorkerConfig] = {}
|
||||
for name, raw_settings in raw_workers.items():
|
||||
if not isinstance(name, str) or not name or name.strip() != name:
|
||||
raise ApplicationConfigLoadError("worker names must be non-empty strings")
|
||||
if not isinstance(raw_settings, dict):
|
||||
raise ApplicationConfigLoadError(f"workers.{name} must be a mapping")
|
||||
unknown = set(raw_settings) - ApplicationConfigLoader._WORKER_FIELDS
|
||||
if unknown:
|
||||
raise ApplicationConfigLoadError(
|
||||
f"workers.{name} contains unsupported fields: {sorted(unknown)}"
|
||||
)
|
||||
workers[name] = ApplicationConfigLoader._worker_config(name, raw_settings)
|
||||
return workers
|
||||
|
||||
@staticmethod
|
||||
def _worker_config(name: str, raw: Mapping[str, Any]) -> WorkerConfig:
|
||||
disabled = raw.get("disabled", False)
|
||||
if not isinstance(disabled, bool):
|
||||
raise ApplicationConfigLoadError(f"workers.{name}.disabled must be boolean")
|
||||
interval = ApplicationConfigLoader._duration(name, "interval_seconds", raw)
|
||||
pause = ApplicationConfigLoader._duration(name, "pause_seconds", raw)
|
||||
config = WorkerConfig(disabled, interval, pause)
|
||||
try:
|
||||
config.validate_schedule()
|
||||
except WorkerConfigurationError as error:
|
||||
raise ApplicationConfigLoadError(f"workers.{name} has an invalid schedule") from error
|
||||
return config
|
||||
|
||||
@staticmethod
|
||||
def _duration(name: str, field: str, raw: Mapping[str, Any]) -> float | None:
|
||||
if field not in raw:
|
||||
return None
|
||||
value = raw[field]
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
raise ApplicationConfigLoadError(f"workers.{name}.{field} must be numeric")
|
||||
duration = float(value)
|
||||
if not math.isfinite(duration):
|
||||
raise ApplicationConfigLoadError(f"workers.{name}.{field} must be finite")
|
||||
return duration
|
||||
@@ -0,0 +1,161 @@
|
||||
import asyncio
|
||||
import math
|
||||
from collections.abc import Awaitable, Callable
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from enum import StrEnum
|
||||
import logging
|
||||
from threading import RLock
|
||||
from typing import Generic, TypeVar
|
||||
|
||||
from plba.config.loader import ApplicationConfigLoadError, ApplicationConfigLoader
|
||||
from plba.config.models import ApplicationConfigSnapshot, LoadedApplicationConfig
|
||||
from plba.runtime.worker.config import WorkerConfig
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class ConfigReloadStatus(StrEnum):
|
||||
UNCHANGED = "unchanged"
|
||||
RELOADED = "reloaded"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ConfigReloadResult(Generic[T]):
|
||||
status: ConfigReloadStatus
|
||||
snapshot: ApplicationConfigSnapshot[T] | None = None
|
||||
error: Exception | None = None
|
||||
|
||||
|
||||
class ApplicationConfigUnavailable(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class ApplicationConfigManager(Generic[T]):
|
||||
def __init__(
|
||||
self, loader: ApplicationConfigLoader[T], logger: logging.Logger | None = None
|
||||
) -> None:
|
||||
self._loader = loader
|
||||
self._logger = logger or logging.getLogger(__name__)
|
||||
self._lock = RLock()
|
||||
self._snapshot: ApplicationConfigSnapshot[T] | None = None
|
||||
self._revision = 0
|
||||
self._last_seen_signature: tuple[int, int, int, int] | None = None
|
||||
|
||||
def load_initial(self) -> ApplicationConfigSnapshot[T]:
|
||||
try:
|
||||
loaded = self._loader.load()
|
||||
signature = self._loader.signature()
|
||||
except Exception as error:
|
||||
self._logger.error(
|
||||
"initial application config load failed",
|
||||
extra={"attributes": {"error_type": type(error).__name__}},
|
||||
)
|
||||
raise
|
||||
with self._lock:
|
||||
self._revision += 1
|
||||
self._snapshot = self._snapshot_from(loaded)
|
||||
self._last_seen_signature = signature
|
||||
snapshot = self.current()
|
||||
self._logger.info(
|
||||
"initial application config loaded",
|
||||
extra={"attributes": {"revision": snapshot.revision}},
|
||||
)
|
||||
return snapshot
|
||||
|
||||
def current(self) -> ApplicationConfigSnapshot[T]:
|
||||
with self._lock:
|
||||
if self._snapshot is None:
|
||||
raise ApplicationConfigUnavailable("application config is not loaded")
|
||||
snapshot = self._snapshot
|
||||
return ApplicationConfigSnapshot(
|
||||
application_id=snapshot.application_id,
|
||||
workers=snapshot.workers,
|
||||
application_config=deepcopy(snapshot.application_config),
|
||||
revision=snapshot.revision,
|
||||
)
|
||||
|
||||
def get_worker_config(self, worker_name: str) -> WorkerConfig:
|
||||
with self._lock:
|
||||
if self._snapshot is None:
|
||||
raise ApplicationConfigUnavailable("application config is not loaded")
|
||||
return self._snapshot.workers.get(worker_name, WorkerConfig())
|
||||
|
||||
def reload_if_changed(self) -> ConfigReloadResult[T]:
|
||||
try:
|
||||
signature = self._loader.signature()
|
||||
except OSError as error:
|
||||
with self._lock:
|
||||
if self._last_seen_signature is None:
|
||||
return ConfigReloadResult(ConfigReloadStatus.UNCHANGED)
|
||||
return self._failure(error, None)
|
||||
with self._lock:
|
||||
if signature == self._last_seen_signature:
|
||||
return ConfigReloadResult(ConfigReloadStatus.UNCHANGED)
|
||||
self._last_seen_signature = signature
|
||||
try:
|
||||
loaded = self._loader.load()
|
||||
except ApplicationConfigLoadError as error:
|
||||
return self._failure(error, signature)
|
||||
with self._lock:
|
||||
self._revision += 1
|
||||
self._snapshot = self._snapshot_from(loaded)
|
||||
snapshot = self.current()
|
||||
self._logger.info(
|
||||
"application config reloaded",
|
||||
extra={"attributes": {"revision": snapshot.revision}},
|
||||
)
|
||||
return ConfigReloadResult(ConfigReloadStatus.RELOADED, snapshot=snapshot)
|
||||
|
||||
async def watch(
|
||||
self,
|
||||
stop_event: asyncio.Event,
|
||||
interval_seconds: float,
|
||||
on_reload_failed: Callable[[Exception], Awaitable[None]],
|
||||
sleep: Callable[[float], Awaitable[None]] = asyncio.sleep,
|
||||
) -> None:
|
||||
if (
|
||||
isinstance(interval_seconds, bool)
|
||||
or not isinstance(interval_seconds, (int, float))
|
||||
or not math.isfinite(interval_seconds)
|
||||
or interval_seconds <= 0
|
||||
):
|
||||
raise ValueError("interval_seconds must be a positive finite number")
|
||||
while not stop_event.is_set():
|
||||
sleep_task = asyncio.ensure_future(sleep(interval_seconds))
|
||||
stop_task = asyncio.create_task(stop_event.wait())
|
||||
done, pending = await asyncio.wait(
|
||||
(sleep_task, stop_task), return_when=asyncio.FIRST_COMPLETED
|
||||
)
|
||||
if sleep_task in done:
|
||||
await sleep_task
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
await asyncio.gather(*pending, return_exceptions=True)
|
||||
if stop_task in done:
|
||||
break
|
||||
result = await asyncio.to_thread(self.reload_if_changed)
|
||||
if result.status is ConfigReloadStatus.FAILED and result.error is not None:
|
||||
await on_reload_failed(result.error)
|
||||
|
||||
def _failure(
|
||||
self, error: Exception, signature: tuple[int, int, int, int] | None
|
||||
) -> ConfigReloadResult[T]:
|
||||
with self._lock:
|
||||
self._last_seen_signature = signature
|
||||
self._logger.error(
|
||||
"application config reload failed",
|
||||
extra={"attributes": {"error_type": type(error).__name__}},
|
||||
)
|
||||
return ConfigReloadResult(ConfigReloadStatus.FAILED, error=error)
|
||||
|
||||
def _snapshot_from(
|
||||
self, loaded: LoadedApplicationConfig[T]
|
||||
) -> ApplicationConfigSnapshot[T]:
|
||||
return ApplicationConfigSnapshot(
|
||||
application_id=loaded.application_id,
|
||||
workers=loaded.workers,
|
||||
application_config=loaded.application_config,
|
||||
revision=self._revision,
|
||||
)
|
||||
@@ -0,0 +1,27 @@
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Generic, TypeVar
|
||||
|
||||
from plba.runtime.worker.config import WorkerConfig
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ApplicationConfigSnapshot(Generic[T]):
|
||||
application_id: str | None
|
||||
workers: Mapping[str, WorkerConfig]
|
||||
application_config: T
|
||||
revision: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "workers", MappingProxyType(dict(self.workers)))
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LoadedApplicationConfig(Generic[T]):
|
||||
application_id: str | None
|
||||
workers: Mapping[str, WorkerConfig]
|
||||
application_config: T
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
from plba.health.aggregator import HealthAggregator, HealthInputInvalid, HealthInputReader
|
||||
from plba.health.contributors import (
|
||||
DuplicateHealthContributorError,
|
||||
HealthContributor,
|
||||
HealthContributorRegistry,
|
||||
)
|
||||
from plba.health.models import (
|
||||
HealthComponent,
|
||||
HealthExpectation,
|
||||
HealthInput,
|
||||
HealthReport,
|
||||
)
|
||||
from plba.health.provider import RuntimeHealthInputProvider
|
||||
|
||||
__all__ = [
|
||||
"DuplicateHealthContributorError",
|
||||
"HealthAggregator",
|
||||
"HealthComponent",
|
||||
"HealthContributor",
|
||||
"HealthContributorRegistry",
|
||||
"HealthExpectation",
|
||||
"HealthInput",
|
||||
"HealthInputInvalid",
|
||||
"HealthInputReader",
|
||||
"HealthReport",
|
||||
"RuntimeHealthInputProvider",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,118 @@
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from typing import Protocol
|
||||
|
||||
from plba.health.models import (
|
||||
HealthComponent,
|
||||
HealthExpectation,
|
||||
HealthInput,
|
||||
HealthReport,
|
||||
)
|
||||
from plba.runtime.worker.models import HealthStatus
|
||||
|
||||
|
||||
class HealthInputInvalid(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class HealthInputReader(Protocol):
|
||||
async def read(self) -> HealthInput: ...
|
||||
|
||||
|
||||
class HealthAggregator:
|
||||
_LIFECYCLES = {"new", "starting", "running", "stopping", "stopped", "failed"}
|
||||
|
||||
def __init__(self, input_provider: HealthInputReader) -> None:
|
||||
self._input_provider = input_provider
|
||||
|
||||
async def aggregate(self) -> HealthReport:
|
||||
health_input = await self._input_provider.read()
|
||||
self._validate(health_input)
|
||||
components = self._fill_missing(health_input)
|
||||
status = self._aggregate_status(health_input.lifecycle, components)
|
||||
return HealthReport(
|
||||
application_id=health_input.application_id,
|
||||
lifecycle=health_input.lifecycle,
|
||||
status=status,
|
||||
components=tuple(sorted(components, key=lambda item: item.name)),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate(health_input: HealthInput) -> None:
|
||||
if not isinstance(health_input, HealthInput):
|
||||
raise HealthInputInvalid("health input has an unsupported type")
|
||||
if not isinstance(health_input.application_id, str) or not health_input.application_id:
|
||||
raise HealthInputInvalid("application_id must be a non-empty string")
|
||||
if not isinstance(health_input.lifecycle, str) or health_input.lifecycle not in HealthAggregator._LIFECYCLES:
|
||||
raise HealthInputInvalid("lifecycle is not supported")
|
||||
if any(not isinstance(item, HealthExpectation) for item in health_input.expected_components):
|
||||
raise HealthInputInvalid("expected component has an unsupported type")
|
||||
if any(not isinstance(item, HealthComponent) for item in health_input.components):
|
||||
raise HealthInputInvalid("component has an unsupported type")
|
||||
expected_names = [item.name for item in health_input.expected_components]
|
||||
component_names = [item.name for item in health_input.components]
|
||||
if any(not isinstance(name, str) for name in expected_names + component_names):
|
||||
raise HealthInputInvalid("component names must be strings")
|
||||
if any(not isinstance(item.critical, bool) for item in health_input.expected_components):
|
||||
raise HealthInputInvalid("expected component critical flags must be boolean")
|
||||
if len(expected_names) != len(set(expected_names)):
|
||||
raise HealthInputInvalid("expected component names must be unique")
|
||||
if len(component_names) != len(set(component_names)):
|
||||
raise HealthInputInvalid("component names must be unique")
|
||||
if any(not name for name in expected_names + component_names):
|
||||
raise HealthInputInvalid("component names must be non-empty")
|
||||
for component in health_input.components:
|
||||
if not isinstance(component.status, HealthStatus):
|
||||
raise HealthInputInvalid(f"invalid status for component {component.name}")
|
||||
if not isinstance(component.critical, bool):
|
||||
raise HealthInputInvalid(f"invalid critical flag for {component.name}")
|
||||
if component.detail is not None and not isinstance(component.detail, str):
|
||||
raise HealthInputInvalid(f"invalid detail for component {component.name}")
|
||||
if not isinstance(component.meta, Mapping):
|
||||
raise HealthInputInvalid(f"invalid meta for component {component.name}")
|
||||
if any(not isinstance(key, str) for key in component.meta):
|
||||
raise HealthInputInvalid(f"meta keys must be strings for {component.name}")
|
||||
try:
|
||||
json.dumps(dict(component.meta), allow_nan=False)
|
||||
except (TypeError, ValueError) as error:
|
||||
raise HealthInputInvalid(
|
||||
f"meta is not JSON serializable for component {component.name}"
|
||||
) from error
|
||||
|
||||
@staticmethod
|
||||
def _fill_missing(health_input: HealthInput) -> list[HealthComponent]:
|
||||
components = {item.name: item for item in health_input.components}
|
||||
for expected in health_input.expected_components:
|
||||
if expected.name not in components:
|
||||
components[expected.name] = HealthComponent(
|
||||
name=expected.name,
|
||||
status=(
|
||||
HealthStatus.UNHEALTHY
|
||||
if expected.critical
|
||||
else HealthStatus.DEGRADED
|
||||
),
|
||||
critical=expected.critical,
|
||||
detail="component has not reported health",
|
||||
)
|
||||
return list(components.values())
|
||||
|
||||
@staticmethod
|
||||
def _aggregate_status(lifecycle: str, components: list[HealthComponent]) -> str:
|
||||
if lifecycle == "stopped":
|
||||
return "stopped"
|
||||
statuses = [HealthStatus.OK]
|
||||
if lifecycle in {"new", "failed"}:
|
||||
statuses.append(HealthStatus.UNHEALTHY)
|
||||
elif lifecycle in {"starting", "stopping"}:
|
||||
statuses.append(HealthStatus.DEGRADED)
|
||||
order = {
|
||||
HealthStatus.OK: 0,
|
||||
HealthStatus.DEGRADED: 1,
|
||||
HealthStatus.UNHEALTHY: 2,
|
||||
}
|
||||
for component in components:
|
||||
status = component.status
|
||||
if status is HealthStatus.UNHEALTHY and not component.critical:
|
||||
status = HealthStatus.DEGRADED
|
||||
statuses.append(status)
|
||||
return max(statuses, key=order.__getitem__).value
|
||||
@@ -0,0 +1,35 @@
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
|
||||
from plba.runtime.worker.models import ComponentHealth
|
||||
|
||||
|
||||
HealthReader = Callable[[], Awaitable[ComponentHealth]]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class HealthContributor:
|
||||
name: str
|
||||
health_reader: HealthReader
|
||||
critical: bool = True
|
||||
|
||||
|
||||
class DuplicateHealthContributorError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class HealthContributorRegistry:
|
||||
def __init__(self) -> None:
|
||||
self._contributors: dict[str, HealthContributor] = {}
|
||||
|
||||
def register(self, contributor: HealthContributor) -> None:
|
||||
if not contributor.name or contributor.name.strip() != contributor.name:
|
||||
raise ValueError("health contributor name must be non-empty")
|
||||
if contributor.name in self._contributors:
|
||||
raise DuplicateHealthContributorError(
|
||||
f"health contributor already registered: {contributor.name}"
|
||||
)
|
||||
self._contributors[contributor.name] = contributor
|
||||
|
||||
def all(self) -> tuple[HealthContributor, ...]:
|
||||
return tuple(self._contributors.values())
|
||||
@@ -0,0 +1,37 @@
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from plba.runtime.worker.models import HealthStatus
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class HealthExpectation:
|
||||
name: str
|
||||
critical: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class HealthComponent:
|
||||
name: str
|
||||
status: HealthStatus
|
||||
critical: bool
|
||||
detail: str | None = None
|
||||
meta: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class HealthInput:
|
||||
application_id: str
|
||||
lifecycle: str
|
||||
expected_components: tuple[HealthExpectation, ...]
|
||||
components: tuple[HealthComponent, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class HealthReport:
|
||||
application_id: str
|
||||
lifecycle: str
|
||||
status: str
|
||||
components: tuple[HealthComponent, ...]
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import asyncio
|
||||
|
||||
from plba.health.contributors import HealthContributorRegistry
|
||||
from plba.health.models import HealthComponent, HealthExpectation, HealthInput
|
||||
from plba.runtime.application import ApplicationRuntime
|
||||
from plba.runtime.worker.models import ComponentHealth
|
||||
from plba.runtime.worker.supervisor import WorkerSupervisor
|
||||
|
||||
|
||||
class RuntimeHealthInputProvider:
|
||||
def __init__(
|
||||
self,
|
||||
runtime: ApplicationRuntime,
|
||||
worker_supervisor: WorkerSupervisor,
|
||||
contributors: HealthContributorRegistry,
|
||||
) -> None:
|
||||
self._runtime = runtime
|
||||
self._worker_supervisor = worker_supervisor
|
||||
self._contributors = contributors
|
||||
|
||||
async def read(self) -> HealthInput:
|
||||
worker_expectations = self._worker_supervisor.expected_workers()
|
||||
contributor_definitions = self._contributors.all()
|
||||
expected = tuple(
|
||||
[HealthExpectation(name, critical) for name, critical in worker_expectations]
|
||||
+ [
|
||||
HealthExpectation(item.name, item.critical)
|
||||
for item in contributor_definitions
|
||||
]
|
||||
)
|
||||
actual = [
|
||||
HealthComponent(
|
||||
name=snapshot.name,
|
||||
status=snapshot.health.status,
|
||||
critical=snapshot.health.critical,
|
||||
detail=snapshot.health.detail,
|
||||
meta=snapshot.health.meta,
|
||||
)
|
||||
for snapshot in self._worker_supervisor.snapshots()
|
||||
]
|
||||
contributor_health = await asyncio.gather(
|
||||
*(item.health_reader() for item in contributor_definitions)
|
||||
)
|
||||
actual.extend(
|
||||
self._component_from_contributor(item.name, item.critical, health)
|
||||
for item, health in zip(contributor_definitions, contributor_health)
|
||||
)
|
||||
return HealthInput(
|
||||
application_id=self._runtime.context.application_id,
|
||||
lifecycle=self._runtime.lifecycle.value,
|
||||
expected_components=expected,
|
||||
components=tuple(actual),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _component_from_contributor(
|
||||
name: str, critical: bool, health: ComponentHealth
|
||||
) -> HealthComponent:
|
||||
return HealthComponent(
|
||||
name=name,
|
||||
status=health.status,
|
||||
critical=critical,
|
||||
detail=health.detail,
|
||||
meta=dict(health.meta),
|
||||
)
|
||||
@@ -0,0 +1,3 @@
|
||||
from plba.http.control_api import ControlApi
|
||||
|
||||
__all__ = ["ControlApi"]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,41 @@
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from plba.http.errors import ControlRequestError
|
||||
|
||||
|
||||
class ASGIHttpTransport:
|
||||
async def read_body(self, receive: Any) -> bytes:
|
||||
chunks: list[bytes] = []
|
||||
while True:
|
||||
message = await receive()
|
||||
if message.get("type") == "http.disconnect":
|
||||
raise ControlRequestError("client disconnected before request completed")
|
||||
if message.get("type") != "http.request":
|
||||
continue
|
||||
chunks.append(message.get("body", b""))
|
||||
if not message.get("more_body", False):
|
||||
return b"".join(chunks)
|
||||
|
||||
async def send_json(self, send: Any, status: int, payload: dict[str, Any]) -> None:
|
||||
body = json.dumps(payload, allow_nan=False).encode("utf-8")
|
||||
await send(
|
||||
{
|
||||
"type": "http.response.start",
|
||||
"status": status,
|
||||
"headers": [
|
||||
(b"content-type", b"application/json; charset=utf-8"),
|
||||
(b"content-length", str(len(body)).encode("ascii")),
|
||||
],
|
||||
}
|
||||
)
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
|
||||
async def handle_lifespan(self, receive: Any, send: Any) -> None:
|
||||
while True:
|
||||
message = await receive()
|
||||
if message.get("type") == "lifespan.startup":
|
||||
await send({"type": "lifespan.startup.complete"})
|
||||
elif message.get("type") == "lifespan.shutdown":
|
||||
await send({"type": "lifespan.shutdown.complete"})
|
||||
return
|
||||
@@ -0,0 +1,81 @@
|
||||
from typing import Any
|
||||
|
||||
from plba.health.aggregator import HealthAggregator, HealthInputInvalid
|
||||
from plba.http.asgi import ASGIHttpTransport
|
||||
from plba.http.errors import ControlRequestError
|
||||
from plba.http.requests import StopRequest
|
||||
from plba.http.serialization import health_report_payload
|
||||
from plba.runtime.application import ApplicationRuntime, RuntimeLifecycleError
|
||||
|
||||
|
||||
class ControlApi:
|
||||
"""ASGI route adapter; it owns no server, socket, or listener."""
|
||||
|
||||
def __init__(
|
||||
self, runtime: ApplicationRuntime, health_aggregator: HealthAggregator
|
||||
) -> None:
|
||||
self._runtime = runtime
|
||||
self._health_aggregator = health_aggregator
|
||||
self._transport = ASGIHttpTransport()
|
||||
|
||||
async def __call__(self, scope: dict[str, Any], receive: Any, send: Any) -> None:
|
||||
scope_type = scope.get("type")
|
||||
if scope_type == "lifespan":
|
||||
await self._transport.handle_lifespan(receive, send)
|
||||
return
|
||||
if scope_type != "http":
|
||||
return
|
||||
try:
|
||||
status, payload = await self._dispatch(scope, receive)
|
||||
except HealthInputInvalid as error:
|
||||
status, payload = 500, self._error("HEALTH_INPUT_INVALID", error)
|
||||
except RuntimeLifecycleError as error:
|
||||
status, payload = 409, self._error("APPLICATION_CONTROL_CONFLICT", error)
|
||||
except ControlRequestError as error:
|
||||
status, payload = 400, self._error("HTTP_CONTROL_REJECTED", error)
|
||||
except Exception as error:
|
||||
status, payload = 500, self._error("CONTROL_API_FAILED", error)
|
||||
await self._transport.send_json(send, status, payload)
|
||||
|
||||
async def _dispatch(self, scope: dict[str, Any], receive: Any) -> tuple[int, dict[str, Any]]:
|
||||
method = scope.get("method", "").upper()
|
||||
path = scope.get("path", "")
|
||||
query = scope.get("query_string", b"")
|
||||
if path in {"/health", "/runtime/status", "/runtime/start", "/runtime/stop"} and query:
|
||||
raise ControlRequestError("query parameters are not accepted")
|
||||
if path == "/health":
|
||||
if method != "GET":
|
||||
return 405, self._error("METHOD_NOT_ALLOWED", "GET required")
|
||||
await self._require_empty_body(receive)
|
||||
report = await self._health_aggregator.aggregate()
|
||||
return 200, health_report_payload(report)
|
||||
if path == "/runtime/start":
|
||||
if method != "POST":
|
||||
return 405, self._error("METHOD_NOT_ALLOWED", "POST required")
|
||||
await self._require_empty_body(receive)
|
||||
result = await self._runtime.start()
|
||||
return 200, {"status": result.lifecycle.value}
|
||||
if path == "/runtime/status":
|
||||
if method != "GET":
|
||||
return 405, self._error("METHOD_NOT_ALLOWED", "GET required")
|
||||
await self._require_empty_body(receive)
|
||||
return 200, {"status": self._runtime.lifecycle.value}
|
||||
if path == "/runtime/stop":
|
||||
if method != "POST":
|
||||
return 405, self._error("METHOD_NOT_ALLOWED", "POST required")
|
||||
request = StopRequest.parse(await self._transport.read_body(receive))
|
||||
result = await self._runtime.stop(
|
||||
force=request.force, timeout_seconds=request.timeout_seconds
|
||||
)
|
||||
status_code = 200 if result.lifecycle.value == "stopped" else 202
|
||||
return status_code, {"status": result.lifecycle.value}
|
||||
return 404, self._error("NOT_FOUND", "control route not found")
|
||||
|
||||
async def _require_empty_body(self, receive: Any) -> None:
|
||||
if await self._transport.read_body(receive):
|
||||
raise ControlRequestError("request body is not accepted")
|
||||
|
||||
@staticmethod
|
||||
def _error(code: str, error: Exception | str) -> dict[str, str]:
|
||||
return {"error": code, "message": str(error)}
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
class ControlRequestError(ValueError):
|
||||
pass
|
||||
@@ -0,0 +1,39 @@
|
||||
import json
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from plba.http.errors import ControlRequestError
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class StopRequest:
|
||||
force: bool = False
|
||||
timeout_seconds: float | None = None
|
||||
|
||||
@classmethod
|
||||
def parse(cls, body: bytes) -> "StopRequest":
|
||||
if not body:
|
||||
raise ControlRequestError("stop request must contain a JSON object")
|
||||
try:
|
||||
decoded: Any = json.loads(body.decode("utf-8"))
|
||||
except (json.JSONDecodeError, UnicodeDecodeError) as error:
|
||||
raise ControlRequestError("stop request must contain valid UTF-8 JSON") from error
|
||||
if not isinstance(decoded, dict):
|
||||
raise ControlRequestError("stop request must be a JSON object")
|
||||
if set(decoded) - {"force", "timeout_seconds"}:
|
||||
raise ControlRequestError("stop request contains unsupported fields")
|
||||
force = decoded.get("force", False)
|
||||
timeout_seconds = decoded.get("timeout_seconds")
|
||||
if not isinstance(force, bool):
|
||||
raise ControlRequestError("force must be a boolean")
|
||||
if "timeout_seconds" in decoded and timeout_seconds is None:
|
||||
raise ControlRequestError("timeout_seconds cannot be null")
|
||||
if timeout_seconds is not None and (
|
||||
isinstance(timeout_seconds, bool)
|
||||
or not isinstance(timeout_seconds, (int, float))
|
||||
or not math.isfinite(timeout_seconds)
|
||||
or timeout_seconds <= 0
|
||||
):
|
||||
raise ControlRequestError("timeout_seconds must be a positive finite number")
|
||||
return cls(force=force, timeout_seconds=timeout_seconds)
|
||||
@@ -0,0 +1,21 @@
|
||||
from typing import Any
|
||||
|
||||
from plba.health.models import HealthReport
|
||||
|
||||
|
||||
def health_report_payload(report: HealthReport) -> dict[str, Any]:
|
||||
return {
|
||||
"application_id": report.application_id,
|
||||
"lifecycle": report.lifecycle,
|
||||
"status": report.status,
|
||||
"components": [
|
||||
{
|
||||
"name": item.name,
|
||||
"status": item.status.value,
|
||||
"critical": item.critical,
|
||||
"detail": item.detail,
|
||||
"meta": dict(item.meta),
|
||||
}
|
||||
for item in report.components
|
||||
],
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
from plba.logging.technical import TechnicalJsonFormatter, create_stdout_handler
|
||||
|
||||
__all__ = ["TechnicalJsonFormatter", "create_stdout_handler"]
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,57 @@
|
||||
from collections.abc import Mapping, Callable
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from typing import Any, TextIO
|
||||
|
||||
from plba.tracing.context import current_business_trace_id
|
||||
|
||||
|
||||
class TechnicalJsonFormatter(logging.Formatter):
|
||||
def __init__(
|
||||
self,
|
||||
application_id: str,
|
||||
trace_id_reader: Callable[[], str | None] = current_business_trace_id,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
if not application_id:
|
||||
raise ValueError("application_id must be non-empty")
|
||||
self._application_id = application_id
|
||||
self._trace_id_reader = trace_id_reader
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
timestamp = datetime.fromtimestamp(record.created, timezone.utc)
|
||||
payload: dict[str, Any] = {
|
||||
"timestamp": timestamp.isoformat(timespec="milliseconds").replace(
|
||||
"+00:00", "Z"
|
||||
),
|
||||
"level": record.levelname,
|
||||
"logger": record.name,
|
||||
"message": record.getMessage(),
|
||||
"application_id": self._application_id,
|
||||
}
|
||||
attributes = getattr(record, "attributes", None)
|
||||
if attributes is not None:
|
||||
if not isinstance(attributes, Mapping) or any(
|
||||
not isinstance(key, str) for key in attributes
|
||||
):
|
||||
raise ValueError("log attributes must be a string-keyed object")
|
||||
payload["attributes"] = dict(attributes)
|
||||
business_trace_id = self._trace_id_reader()
|
||||
if business_trace_id is not None:
|
||||
payload["business_trace_id"] = business_trace_id
|
||||
try:
|
||||
return json.dumps(
|
||||
payload, ensure_ascii=False, allow_nan=False, separators=(",", ":")
|
||||
)
|
||||
except (TypeError, ValueError) as error:
|
||||
raise ValueError("technical log attributes must be JSON-safe") from error
|
||||
|
||||
|
||||
def create_stdout_handler(
|
||||
application_id: str, stream: TextIO | None = None
|
||||
) -> logging.StreamHandler[TextIO]:
|
||||
handler = logging.StreamHandler(stream if stream is not None else sys.stdout)
|
||||
handler.setFormatter(TechnicalJsonFormatter(application_id))
|
||||
return handler
|
||||
@@ -0,0 +1,25 @@
|
||||
from plba.runtime.application import (
|
||||
ApplicationLifecycle,
|
||||
ApplicationRuntime,
|
||||
ApplicationStartResult,
|
||||
ApplicationStopResult,
|
||||
RuntimeLifecycleError,
|
||||
)
|
||||
from plba.runtime.application_identity import (
|
||||
ApplicationIdentityResolver,
|
||||
ProjectNameReader,
|
||||
TomlProjectNameReader,
|
||||
)
|
||||
from plba.runtime.context import RuntimeContext
|
||||
|
||||
__all__ = [
|
||||
"ApplicationLifecycle",
|
||||
"ApplicationIdentityResolver",
|
||||
"ApplicationRuntime",
|
||||
"ApplicationStartResult",
|
||||
"ApplicationStopResult",
|
||||
"ProjectNameReader",
|
||||
"RuntimeContext",
|
||||
"RuntimeLifecycleError",
|
||||
"TomlProjectNameReader",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,88 @@
|
||||
import logging
|
||||
from plba.runtime.context import RuntimeContext
|
||||
from plba.runtime.lifecycle import (
|
||||
ApplicationLifecycle,
|
||||
ApplicationStartResult,
|
||||
ApplicationStopResult,
|
||||
RuntimeLifecycleError,
|
||||
)
|
||||
from plba.runtime.stop_coordinator import ApplicationStopCoordinator
|
||||
from plba.runtime.worker.supervisor import WorkerSupervisor
|
||||
|
||||
|
||||
class ApplicationRuntime:
|
||||
def __init__(
|
||||
self,
|
||||
context: RuntimeContext,
|
||||
worker_supervisor: WorkerSupervisor,
|
||||
logger: logging.Logger | None = None,
|
||||
) -> None:
|
||||
self.context = context
|
||||
self._worker_supervisor = worker_supervisor
|
||||
self._logger = logger or logging.getLogger(__name__)
|
||||
self._lifecycle = ApplicationLifecycle.NEW
|
||||
self._stop_coordinator = ApplicationStopCoordinator(
|
||||
worker_supervisor,
|
||||
lambda: self._lifecycle,
|
||||
self._set_lifecycle,
|
||||
context.application_id,
|
||||
self._logger,
|
||||
)
|
||||
|
||||
@property
|
||||
def lifecycle(self) -> ApplicationLifecycle:
|
||||
return self._lifecycle
|
||||
|
||||
async def start(self) -> ApplicationStartResult:
|
||||
if self._lifecycle not in {
|
||||
ApplicationLifecycle.NEW,
|
||||
ApplicationLifecycle.STOPPED,
|
||||
}:
|
||||
raise RuntimeLifecycleError(
|
||||
f"application cannot start from {self._lifecycle.value}"
|
||||
)
|
||||
self._lifecycle = ApplicationLifecycle.STARTING
|
||||
self._logger.info(
|
||||
"application start requested",
|
||||
extra={"attributes": {"application_id": self.context.application_id}},
|
||||
)
|
||||
try:
|
||||
workers = await self._worker_supervisor.start_all()
|
||||
except Exception as error:
|
||||
self._lifecycle = ApplicationLifecycle.FAILED
|
||||
self._logger.error(
|
||||
"application start failed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"application_id": self.context.application_id,
|
||||
"error_type": type(error).__name__,
|
||||
}
|
||||
},
|
||||
)
|
||||
raise
|
||||
self._lifecycle = (
|
||||
ApplicationLifecycle.RUNNING
|
||||
if workers.successful
|
||||
else ApplicationLifecycle.FAILED
|
||||
)
|
||||
self._logger.log(
|
||||
logging.INFO if workers.successful else logging.ERROR,
|
||||
"application start completed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"application_id": self.context.application_id,
|
||||
"lifecycle": self._lifecycle.value,
|
||||
"started_workers": list(workers.started),
|
||||
"failed_workers": list(workers.failures),
|
||||
}
|
||||
},
|
||||
)
|
||||
return ApplicationStartResult(self._lifecycle, workers)
|
||||
|
||||
async def stop(
|
||||
self, force: bool = False, timeout_seconds: float | None = None
|
||||
) -> ApplicationStopResult:
|
||||
return await self._stop_coordinator.stop(force, timeout_seconds)
|
||||
|
||||
def _set_lifecycle(self, lifecycle: ApplicationLifecycle) -> None:
|
||||
self._lifecycle = lifecycle
|
||||
@@ -0,0 +1,54 @@
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
import tomllib
|
||||
from typing import Protocol
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from plba.runtime.context import RuntimeContext
|
||||
|
||||
|
||||
class ProjectNameReader(Protocol):
|
||||
def read_project_name(self) -> str | None: ...
|
||||
|
||||
|
||||
class TomlProjectNameReader:
|
||||
def __init__(self, metadata_path: Path) -> None:
|
||||
self._metadata_path = metadata_path
|
||||
|
||||
def read_project_name(self) -> str | None:
|
||||
try:
|
||||
with self._metadata_path.open("rb") as metadata_file:
|
||||
metadata = tomllib.load(metadata_file)
|
||||
except (OSError, tomllib.TOMLDecodeError):
|
||||
return None
|
||||
|
||||
project = metadata.get("project")
|
||||
if not isinstance(project, dict):
|
||||
return None
|
||||
name = project.get("name")
|
||||
return name if isinstance(name, str) and name else None
|
||||
|
||||
|
||||
class ApplicationIdentityResolver:
|
||||
def __init__(
|
||||
self,
|
||||
project_name_reader: ProjectNameReader,
|
||||
uuid_factory: Callable[[], UUID] = uuid4,
|
||||
) -> None:
|
||||
self._project_name_reader = project_name_reader
|
||||
self._uuid_factory = uuid_factory
|
||||
|
||||
def resolve(self, configured_application_id: str | None = None) -> RuntimeContext:
|
||||
if configured_application_id is not None and not isinstance(
|
||||
configured_application_id, str
|
||||
):
|
||||
raise TypeError("configured_application_id must be a string or None")
|
||||
|
||||
if configured_application_id:
|
||||
application_id = configured_application_id
|
||||
else:
|
||||
application_id = self._project_name_reader.read_project_name()
|
||||
if not application_id:
|
||||
application_id = str(self._uuid_factory())
|
||||
|
||||
return RuntimeContext(application_id=application_id)
|
||||
@@ -0,0 +1,8 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RuntimeContext:
|
||||
"""Process-wide identity shared with runtime components."""
|
||||
|
||||
application_id: str
|
||||
@@ -0,0 +1,32 @@
|
||||
from dataclasses import dataclass
|
||||
from enum import StrEnum
|
||||
|
||||
from plba.runtime.worker.supervisor import (
|
||||
WorkerGroupStartResult,
|
||||
WorkerGroupStopResult,
|
||||
)
|
||||
|
||||
|
||||
class ApplicationLifecycle(StrEnum):
|
||||
NEW = "new"
|
||||
STARTING = "starting"
|
||||
RUNNING = "running"
|
||||
STOPPING = "stopping"
|
||||
STOPPED = "stopped"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
class RuntimeLifecycleError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ApplicationStartResult:
|
||||
lifecycle: ApplicationLifecycle
|
||||
workers: WorkerGroupStartResult
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ApplicationStopResult:
|
||||
lifecycle: ApplicationLifecycle
|
||||
workers: WorkerGroupStopResult
|
||||
@@ -0,0 +1,126 @@
|
||||
import asyncio
|
||||
from collections.abc import Callable
|
||||
import logging
|
||||
import math
|
||||
|
||||
from plba.runtime.lifecycle import (
|
||||
ApplicationLifecycle,
|
||||
ApplicationStopResult,
|
||||
RuntimeLifecycleError,
|
||||
)
|
||||
from plba.runtime.worker.supervisor import WorkerGroupStopResult, WorkerSupervisor
|
||||
|
||||
|
||||
class ApplicationStopCoordinator:
|
||||
def __init__(
|
||||
self,
|
||||
worker_supervisor: WorkerSupervisor,
|
||||
lifecycle_reader: Callable[[], ApplicationLifecycle],
|
||||
lifecycle_writer: Callable[[ApplicationLifecycle], None],
|
||||
application_id: str,
|
||||
logger: logging.Logger,
|
||||
) -> None:
|
||||
self._worker_supervisor = worker_supervisor
|
||||
self._lifecycle_reader = lifecycle_reader
|
||||
self._lifecycle_writer = lifecycle_writer
|
||||
self._application_id = application_id
|
||||
self._logger = logger
|
||||
self._stop_task: asyncio.Task[WorkerGroupStopResult] | None = None
|
||||
|
||||
async def stop(
|
||||
self, force: bool = False, timeout_seconds: float | None = None
|
||||
) -> ApplicationStopResult:
|
||||
self._validate_options(force, timeout_seconds)
|
||||
lifecycle = self._lifecycle_reader()
|
||||
if lifecycle is ApplicationLifecycle.STOPPED:
|
||||
return ApplicationStopResult(lifecycle, WorkerGroupStopResult((), ()))
|
||||
if lifecycle is ApplicationLifecycle.STOPPING:
|
||||
if force:
|
||||
self._worker_supervisor.request_stop_all(force=True)
|
||||
else:
|
||||
self._lifecycle_writer(ApplicationLifecycle.STOPPING)
|
||||
self._logger.info(
|
||||
"application stop requested",
|
||||
extra={"attributes": {"application_id": self._application_id, "force": force}},
|
||||
)
|
||||
self._stop_task = asyncio.create_task(
|
||||
self._finish_stop(force), name="application:stop"
|
||||
)
|
||||
|
||||
if self._stop_task is None:
|
||||
raise RuntimeLifecycleError("application stop operation was not created")
|
||||
if timeout_seconds is None:
|
||||
workers = await asyncio.shield(self._stop_task)
|
||||
return ApplicationStopResult(self._lifecycle_reader(), workers)
|
||||
done, _ = await asyncio.wait({self._stop_task}, timeout=timeout_seconds)
|
||||
if not done:
|
||||
return self._stopping_result()
|
||||
return ApplicationStopResult(self._lifecycle_reader(), self._stop_task.result())
|
||||
|
||||
async def _finish_stop(self, force: bool) -> WorkerGroupStopResult:
|
||||
try:
|
||||
workers = await self._worker_supervisor.stop_all(force=force)
|
||||
except Exception as error:
|
||||
self._lifecycle_writer(ApplicationLifecycle.FAILED)
|
||||
self._logger.error(
|
||||
"application stop failed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"application_id": self._application_id,
|
||||
"error_type": type(error).__name__,
|
||||
}
|
||||
},
|
||||
)
|
||||
raise
|
||||
lifecycle = (
|
||||
ApplicationLifecycle.STOPPED
|
||||
if workers.complete
|
||||
else ApplicationLifecycle.STOPPING
|
||||
)
|
||||
self._lifecycle_writer(lifecycle)
|
||||
self._logger.info(
|
||||
"application stop completed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"application_id": self._application_id,
|
||||
"lifecycle": lifecycle.value,
|
||||
"stopped_workers": list(workers.stopped),
|
||||
}
|
||||
},
|
||||
)
|
||||
return workers
|
||||
|
||||
def _stopping_result(self) -> ApplicationStopResult:
|
||||
stopping = tuple(
|
||||
sorted(
|
||||
snapshot.name
|
||||
for snapshot in self._worker_supervisor.snapshots()
|
||||
if snapshot.lifecycle.value in {"starting", "running", "stopping"}
|
||||
)
|
||||
)
|
||||
self._logger.warning(
|
||||
"application stop is continuing",
|
||||
extra={
|
||||
"attributes": {
|
||||
"application_id": self._application_id,
|
||||
"workers_stopping": list(stopping),
|
||||
}
|
||||
},
|
||||
)
|
||||
return ApplicationStopResult(
|
||||
ApplicationLifecycle.STOPPING, WorkerGroupStopResult((), stopping)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_options(force: bool, timeout_seconds: float | None) -> None:
|
||||
if not isinstance(force, bool):
|
||||
raise ValueError("force must be a boolean")
|
||||
if timeout_seconds is None:
|
||||
return
|
||||
if (
|
||||
isinstance(timeout_seconds, bool)
|
||||
or not isinstance(timeout_seconds, (int, float))
|
||||
or not math.isfinite(timeout_seconds)
|
||||
or timeout_seconds <= 0
|
||||
):
|
||||
raise ValueError("timeout_seconds must be a positive finite number")
|
||||
@@ -0,0 +1,29 @@
|
||||
from plba.runtime.worker.config import WorkerConfig, WorkerConfigReader
|
||||
from plba.runtime.worker.models import (
|
||||
ComponentHealth,
|
||||
HealthStatus,
|
||||
WorkerDefinition,
|
||||
WorkerLifecycle,
|
||||
WorkerSnapshot,
|
||||
)
|
||||
from plba.runtime.worker.registry import DuplicateWorkerError, WorkerRegistry
|
||||
from plba.runtime.worker.supervisor import (
|
||||
WorkerGroupStartResult,
|
||||
WorkerGroupStopResult,
|
||||
WorkerSupervisor,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ComponentHealth",
|
||||
"DuplicateWorkerError",
|
||||
"HealthStatus",
|
||||
"WorkerConfig",
|
||||
"WorkerConfigReader",
|
||||
"WorkerDefinition",
|
||||
"WorkerGroupStartResult",
|
||||
"WorkerGroupStopResult",
|
||||
"WorkerLifecycle",
|
||||
"WorkerRegistry",
|
||||
"WorkerSnapshot",
|
||||
"WorkerSupervisor",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,33 @@
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
from typing import Protocol
|
||||
|
||||
|
||||
class WorkerConfigurationError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkerConfig:
|
||||
disabled: bool = False
|
||||
interval_seconds: float | None = None
|
||||
pause_seconds: float | None = None
|
||||
|
||||
def validate_schedule(self) -> None:
|
||||
if self.disabled:
|
||||
return
|
||||
has_interval = self.interval_seconds is not None
|
||||
has_pause = self.pause_seconds is not None
|
||||
if has_interval == has_pause:
|
||||
raise WorkerConfigurationError(
|
||||
"exactly one of interval_seconds or pause_seconds is required"
|
||||
)
|
||||
duration = self.interval_seconds if has_interval else self.pause_seconds
|
||||
if isinstance(duration, bool) or not isinstance(duration, (int, float)):
|
||||
raise WorkerConfigurationError("worker schedule must be a number")
|
||||
if not math.isfinite(duration) or duration <= 0:
|
||||
raise WorkerConfigurationError("worker schedule must be positive")
|
||||
|
||||
|
||||
class WorkerConfigReader(Protocol):
|
||||
def get_worker_config(self, worker_name: str) -> WorkerConfig: ...
|
||||
@@ -0,0 +1,87 @@
|
||||
from plba.runtime.worker.models import (
|
||||
ComponentHealth,
|
||||
HealthStatus,
|
||||
WorkerDefinition,
|
||||
)
|
||||
|
||||
|
||||
class WorkerHealthTracker:
|
||||
def __init__(self, definition: WorkerDefinition) -> None:
|
||||
self._critical = definition.critical
|
||||
self._health_check = definition.health_check
|
||||
self._automatic_status = HealthStatus.OK
|
||||
self._application_status = HealthStatus.OK
|
||||
self._application_detail: str | None = None
|
||||
self._application_meta: dict[str, object] = {}
|
||||
self._last_error: BaseException | None = None
|
||||
|
||||
@property
|
||||
def last_error(self) -> BaseException | None:
|
||||
return self._last_error
|
||||
|
||||
def cycle_succeeded(self) -> None:
|
||||
self._last_error = None
|
||||
self._automatic_status = HealthStatus.OK
|
||||
|
||||
def cycle_failed(self, error: BaseException) -> None:
|
||||
self._last_error = error
|
||||
self._automatic_status = HealthStatus.UNHEALTHY
|
||||
|
||||
def runtime_failed(self, error: BaseException) -> None:
|
||||
self._last_error = error
|
||||
self._automatic_status = HealthStatus.UNHEALTHY
|
||||
|
||||
def cycle_exceeded_interval(self) -> None:
|
||||
self._automatic_status = HealthStatus.DEGRADED
|
||||
|
||||
async def refresh_application_health(self) -> None:
|
||||
if self._health_check is None:
|
||||
self._application_status = HealthStatus.OK
|
||||
self._application_detail = None
|
||||
self._application_meta = {}
|
||||
return
|
||||
try:
|
||||
health = await self._health_check()
|
||||
except Exception as error:
|
||||
self._application_status = HealthStatus.UNHEALTHY
|
||||
self._application_detail = f"health check failed: {error}"
|
||||
self._application_meta = {}
|
||||
return
|
||||
if not isinstance(health, ComponentHealth) or not isinstance(
|
||||
health.status, HealthStatus
|
||||
):
|
||||
self._application_status = HealthStatus.UNHEALTHY
|
||||
self._application_detail = "health check returned an invalid status"
|
||||
self._application_meta = {}
|
||||
return
|
||||
self._application_status = health.status
|
||||
self._application_detail = health.detail
|
||||
self._application_meta = dict(health.meta)
|
||||
|
||||
def snapshot(self) -> ComponentHealth:
|
||||
status_order = {
|
||||
HealthStatus.OK: 0,
|
||||
HealthStatus.DEGRADED: 1,
|
||||
HealthStatus.UNHEALTHY: 2,
|
||||
}
|
||||
status = max(
|
||||
(self._automatic_status, self._application_status),
|
||||
key=status_order.__getitem__,
|
||||
)
|
||||
details = self._details()
|
||||
return ComponentHealth(
|
||||
status=status,
|
||||
critical=self._critical,
|
||||
detail="; ".join(details) or None,
|
||||
meta=self._application_meta,
|
||||
)
|
||||
|
||||
def _details(self) -> list[str]:
|
||||
details: list[str] = []
|
||||
if self._automatic_status is HealthStatus.UNHEALTHY and self._last_error:
|
||||
details.append(f"worker cycle failed: {self._last_error}")
|
||||
elif self._automatic_status is HealthStatus.DEGRADED:
|
||||
details.append("worker cycle exceeded interval_seconds")
|
||||
if self._application_detail:
|
||||
details.append(self._application_detail)
|
||||
return details
|
||||
@@ -0,0 +1,48 @@
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from enum import StrEnum
|
||||
from typing import Any
|
||||
|
||||
|
||||
class HealthStatus(StrEnum):
|
||||
OK = "ok"
|
||||
DEGRADED = "degraded"
|
||||
UNHEALTHY = "unhealthy"
|
||||
|
||||
|
||||
class WorkerLifecycle(StrEnum):
|
||||
NEW = "new"
|
||||
STARTING = "starting"
|
||||
RUNNING = "running"
|
||||
STOPPING = "stopping"
|
||||
STOPPED = "stopped"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ComponentHealth:
|
||||
status: HealthStatus = HealthStatus.OK
|
||||
critical: bool = True
|
||||
detail: str | None = None
|
||||
meta: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
HealthCheck = Callable[[], Awaitable[ComponentHealth]]
|
||||
WorkerProcess = Callable[[], Awaitable[None]]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkerDefinition:
|
||||
name: str
|
||||
process: WorkerProcess
|
||||
critical: bool = True
|
||||
health_check: HealthCheck | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkerSnapshot:
|
||||
name: str
|
||||
lifecycle: WorkerLifecycle
|
||||
health: ComponentHealth
|
||||
last_error: BaseException | None = None
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
from plba.runtime.worker.models import WorkerDefinition
|
||||
|
||||
|
||||
class DuplicateWorkerError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class WorkerRegistry:
|
||||
def __init__(self) -> None:
|
||||
self._definitions: dict[str, WorkerDefinition] = {}
|
||||
|
||||
def register(self, definition: WorkerDefinition) -> None:
|
||||
if not definition.name or definition.name.strip() != definition.name:
|
||||
raise ValueError("worker name must be non-empty and have no outer spaces")
|
||||
if definition.name in self._definitions:
|
||||
raise DuplicateWorkerError(f"worker already registered: {definition.name}")
|
||||
self._definitions[definition.name] = definition
|
||||
|
||||
def get(self, name: str) -> WorkerDefinition:
|
||||
return self._definitions[name]
|
||||
|
||||
def all(self) -> tuple[WorkerDefinition, ...]:
|
||||
return tuple(self._definitions.values())
|
||||
@@ -0,0 +1,153 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from plba.runtime.worker.config import WorkerConfig, WorkerConfigReader
|
||||
from plba.runtime.worker.health import WorkerHealthTracker
|
||||
from plba.runtime.worker.models import (
|
||||
WorkerDefinition,
|
||||
WorkerLifecycle,
|
||||
WorkerSnapshot,
|
||||
)
|
||||
from plba.runtime.worker.schedule import WorkerScheduleController
|
||||
|
||||
AsyncSleep = Callable[[float], Awaitable[None]]
|
||||
Clock = Callable[[], float]
|
||||
|
||||
|
||||
class WorkerRunner:
|
||||
def __init__(
|
||||
self,
|
||||
definition: WorkerDefinition,
|
||||
config_reader: WorkerConfigReader,
|
||||
sleep: AsyncSleep = asyncio.sleep,
|
||||
clock: Clock = time.monotonic,
|
||||
logger: logging.Logger | None = None,
|
||||
) -> None:
|
||||
self._definition = definition
|
||||
self._config_reader = config_reader
|
||||
self._sleep = sleep
|
||||
self._clock = clock
|
||||
self._logger = logger or logging.getLogger(__name__)
|
||||
self._schedule = WorkerScheduleController(definition.name, config_reader, sleep)
|
||||
self._lifecycle = WorkerLifecycle.NEW
|
||||
self._health = WorkerHealthTracker(definition)
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
self._stop_requested = asyncio.Event()
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return self._definition.name
|
||||
|
||||
@property
|
||||
def task(self) -> asyncio.Task[None] | None:
|
||||
return self._task
|
||||
|
||||
def start(self, config: WorkerConfig) -> None:
|
||||
self._schedule.start(config)
|
||||
self._lifecycle = WorkerLifecycle.STARTING
|
||||
self._task = asyncio.create_task(self._run(), name=f"worker:{self.name}")
|
||||
self._logger.info(
|
||||
"worker started", extra={"attributes": {"worker": self.name}}
|
||||
)
|
||||
|
||||
def request_stop(self, force: bool = False) -> None:
|
||||
if self._task is None or self._task.done():
|
||||
if self._lifecycle is not WorkerLifecycle.FAILED:
|
||||
self._lifecycle = WorkerLifecycle.STOPPED
|
||||
return
|
||||
self._lifecycle = WorkerLifecycle.STOPPING
|
||||
self._stop_requested.set()
|
||||
if force:
|
||||
self._task.cancel()
|
||||
|
||||
async def wait_stopped(self) -> None:
|
||||
if self._task is None:
|
||||
self._lifecycle = WorkerLifecycle.STOPPED
|
||||
return
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
self._lifecycle = WorkerLifecycle.STOPPED
|
||||
self._logger.info(
|
||||
"worker force-stopped", extra={"attributes": {"worker": self.name}}
|
||||
)
|
||||
except Exception as error:
|
||||
self._health.runtime_failed(error)
|
||||
self._lifecycle = WorkerLifecycle.FAILED
|
||||
else:
|
||||
if self._lifecycle is not WorkerLifecycle.FAILED:
|
||||
self._lifecycle = WorkerLifecycle.STOPPED
|
||||
|
||||
def snapshot(self) -> WorkerSnapshot:
|
||||
return WorkerSnapshot(
|
||||
name=self.name,
|
||||
lifecycle=self._lifecycle,
|
||||
health=self._health.snapshot(),
|
||||
last_error=self._health.last_error,
|
||||
)
|
||||
|
||||
async def _run(self) -> None:
|
||||
self._lifecycle = WorkerLifecycle.RUNNING
|
||||
try:
|
||||
while not self._stop_requested.is_set():
|
||||
cycle_config = self._schedule.current
|
||||
started_at = self._clock()
|
||||
self._logger.debug(
|
||||
"worker cycle started", extra={"attributes": {"worker": self.name}}
|
||||
)
|
||||
try:
|
||||
await self._definition.process()
|
||||
self._health.cycle_succeeded()
|
||||
self._logger.debug(
|
||||
"worker cycle completed",
|
||||
extra={"attributes": {"worker": self.name}},
|
||||
)
|
||||
except Exception as error:
|
||||
self._health.cycle_failed(error)
|
||||
self._logger.error(
|
||||
"worker cycle failed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"worker": self.name,
|
||||
"error_type": type(error).__name__,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
elapsed = max(0.0, self._clock() - started_at)
|
||||
await self._health.refresh_application_health()
|
||||
config = self._schedule.next_schedule()
|
||||
if self._health.last_error is None:
|
||||
if (
|
||||
cycle_config.interval_seconds is not None
|
||||
and elapsed > cycle_config.interval_seconds
|
||||
):
|
||||
self._health.cycle_exceeded_interval()
|
||||
if self._stop_requested.is_set():
|
||||
break
|
||||
await self._schedule.wait_for_next_cycle(
|
||||
config, elapsed, self._stop_requested
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as error:
|
||||
self._health.runtime_failed(error)
|
||||
self._lifecycle = WorkerLifecycle.FAILED
|
||||
self._logger.error(
|
||||
"worker runtime failed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"worker": self.name,
|
||||
"error_type": type(error).__name__,
|
||||
}
|
||||
},
|
||||
)
|
||||
return
|
||||
self._lifecycle = WorkerLifecycle.STOPPED
|
||||
self._logger.info(
|
||||
"worker stopped", extra={"attributes": {"worker": self.name}}
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import replace
|
||||
|
||||
from plba.runtime.worker.config import WorkerConfig, WorkerConfigReader
|
||||
|
||||
AsyncSleep = Callable[[float], Awaitable[None]]
|
||||
|
||||
|
||||
class WorkerScheduleController:
|
||||
def __init__(
|
||||
self,
|
||||
worker_name: str,
|
||||
config_reader: WorkerConfigReader,
|
||||
sleep: AsyncSleep,
|
||||
) -> None:
|
||||
self._worker_name = worker_name
|
||||
self._config_reader = config_reader
|
||||
self._sleep = sleep
|
||||
self._current: WorkerConfig | None = None
|
||||
|
||||
@property
|
||||
def current(self) -> WorkerConfig:
|
||||
if self._current is None:
|
||||
raise RuntimeError("worker schedule has not started")
|
||||
return self._current
|
||||
|
||||
def start(self, config: WorkerConfig) -> None:
|
||||
if config.disabled:
|
||||
raise ValueError(f"worker is disabled: {self._worker_name}")
|
||||
config.validate_schedule()
|
||||
self._current = config
|
||||
|
||||
def next_schedule(self) -> WorkerConfig:
|
||||
config = self._config_reader.get_worker_config(self._worker_name)
|
||||
if config.disabled and config.interval_seconds is None and config.pause_seconds is None:
|
||||
return self.current
|
||||
config = replace(config, disabled=False)
|
||||
config.validate_schedule()
|
||||
self._current = config
|
||||
return config
|
||||
|
||||
async def wait_for_next_cycle(
|
||||
self, config: WorkerConfig, elapsed: float, stop_requested: asyncio.Event
|
||||
) -> None:
|
||||
delay = self._delay(config, elapsed)
|
||||
if not delay or stop_requested.is_set():
|
||||
return
|
||||
sleep_task = asyncio.ensure_future(self._sleep(delay))
|
||||
stop_task = asyncio.create_task(stop_requested.wait())
|
||||
done, pending = await asyncio.wait(
|
||||
(sleep_task, stop_task), return_when=asyncio.FIRST_COMPLETED
|
||||
)
|
||||
if sleep_task in done:
|
||||
await sleep_task
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
await asyncio.gather(*pending, return_exceptions=True)
|
||||
|
||||
@staticmethod
|
||||
def _delay(config: WorkerConfig, elapsed: float) -> float:
|
||||
if config.pause_seconds is not None:
|
||||
return config.pause_seconds
|
||||
return max(0.0, (config.interval_seconds or 0.0) - elapsed)
|
||||
@@ -0,0 +1,151 @@
|
||||
import asyncio
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from types import MappingProxyType
|
||||
from typing import Mapping
|
||||
|
||||
from plba.runtime.worker.config import WorkerConfigReader
|
||||
from plba.runtime.worker.models import (
|
||||
ComponentHealth,
|
||||
HealthStatus,
|
||||
WorkerLifecycle,
|
||||
WorkerSnapshot,
|
||||
)
|
||||
from plba.runtime.worker.registry import WorkerRegistry
|
||||
from plba.runtime.worker.runner import AsyncSleep, Clock, WorkerRunner
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkerGroupStartResult:
|
||||
started: tuple[str, ...]
|
||||
failures: Mapping[str, BaseException] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def successful(self) -> bool:
|
||||
return not self.failures
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkerGroupStopResult:
|
||||
stopped: tuple[str, ...]
|
||||
stopping: tuple[str, ...]
|
||||
|
||||
@property
|
||||
def complete(self) -> bool:
|
||||
return not self.stopping
|
||||
|
||||
|
||||
class WorkerSupervisor:
|
||||
def __init__(
|
||||
self,
|
||||
registry: WorkerRegistry,
|
||||
config_reader: WorkerConfigReader,
|
||||
sleep: AsyncSleep = asyncio.sleep,
|
||||
clock: Clock | None = None,
|
||||
logger: logging.Logger | None = None,
|
||||
) -> None:
|
||||
self._registry = registry
|
||||
self._config_reader = config_reader
|
||||
self._sleep = sleep
|
||||
self._clock = clock
|
||||
self._logger = logger or logging.getLogger(__name__)
|
||||
self._runners: dict[str, WorkerRunner] = {}
|
||||
self._start_failures: dict[str, BaseException] = {}
|
||||
self._expected_names: set[str] = set()
|
||||
self._started_once = False
|
||||
|
||||
async def start_all(self) -> WorkerGroupStartResult:
|
||||
started: list[str] = []
|
||||
self._start_failures = {}
|
||||
self._expected_names = set()
|
||||
self._started_once = True
|
||||
for definition in self._registry.all():
|
||||
try:
|
||||
config = self._config_reader.get_worker_config(definition.name)
|
||||
if config.disabled:
|
||||
self._runners.pop(definition.name, None)
|
||||
self._logger.info(
|
||||
"worker disabled for application start",
|
||||
extra={"attributes": {"worker": definition.name}},
|
||||
)
|
||||
continue
|
||||
self._expected_names.add(definition.name)
|
||||
runner = WorkerRunner(
|
||||
definition,
|
||||
self._config_reader,
|
||||
sleep=self._sleep,
|
||||
**({"clock": self._clock} if self._clock is not None else {}),
|
||||
logger=self._logger,
|
||||
)
|
||||
runner.start(config)
|
||||
self._runners[definition.name] = runner
|
||||
started.append(definition.name)
|
||||
except Exception as error:
|
||||
self._expected_names.add(definition.name)
|
||||
self._start_failures[definition.name] = error
|
||||
self._logger.error(
|
||||
"worker start failed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"worker": definition.name,
|
||||
"error_type": type(error).__name__,
|
||||
}
|
||||
},
|
||||
)
|
||||
return WorkerGroupStartResult(
|
||||
started=tuple(started),
|
||||
failures=MappingProxyType(dict(self._start_failures)),
|
||||
)
|
||||
|
||||
def snapshots(self) -> tuple[WorkerSnapshot, ...]:
|
||||
snapshots = {name: runner.snapshot() for name, runner in self._runners.items()}
|
||||
for name, error in self._start_failures.items():
|
||||
definition = self._registry.get(name)
|
||||
snapshots[name] = WorkerSnapshot(
|
||||
name=name,
|
||||
lifecycle=WorkerLifecycle.FAILED,
|
||||
health=ComponentHealth(
|
||||
HealthStatus.UNHEALTHY, critical=definition.critical
|
||||
),
|
||||
last_error=error,
|
||||
)
|
||||
return tuple(snapshots[name] for name in sorted(snapshots))
|
||||
|
||||
def expected_workers(self) -> tuple[tuple[str, bool], ...]:
|
||||
if self._started_once:
|
||||
names = self._expected_names
|
||||
else:
|
||||
names = {
|
||||
definition.name
|
||||
for definition in self._registry.all()
|
||||
if not self._config_reader.get_worker_config(definition.name).disabled
|
||||
}
|
||||
return tuple(
|
||||
(name, self._registry.get(name).critical) for name in sorted(names)
|
||||
)
|
||||
|
||||
async def stop_all(
|
||||
self, force: bool = False, timeout_seconds: float | None = None
|
||||
) -> WorkerGroupStopResult:
|
||||
if timeout_seconds is not None and timeout_seconds <= 0:
|
||||
raise ValueError("timeout_seconds must be positive")
|
||||
runners = tuple(self._runners.values())
|
||||
self.request_stop_all(force=force)
|
||||
|
||||
tasks = {runner.task: runner for runner in runners if runner.task is not None}
|
||||
if not tasks:
|
||||
return WorkerGroupStopResult((), ())
|
||||
done, pending = await asyncio.wait(
|
||||
tasks,
|
||||
timeout=timeout_seconds,
|
||||
return_when=asyncio.ALL_COMPLETED,
|
||||
)
|
||||
for task in done:
|
||||
await tasks[task].wait_stopped()
|
||||
stopped = tuple(sorted(tasks[task].name for task in done))
|
||||
stopping = tuple(sorted(tasks[task].name for task in pending))
|
||||
return WorkerGroupStopResult(stopped, stopping)
|
||||
|
||||
def request_stop_all(self, force: bool = False) -> None:
|
||||
for runner in tuple(self._runners.values()):
|
||||
runner.request_stop(force=force)
|
||||
@@ -0,0 +1,19 @@
|
||||
from plba.settings.models import (
|
||||
SettingInvalidError,
|
||||
SettingKey,
|
||||
SettingNotFoundError,
|
||||
SettingsDatabaseUnavailableError,
|
||||
)
|
||||
from plba.settings.postgres import PostgresSettingsRepository
|
||||
from plba.settings.reader import SettingsReader
|
||||
from plba.settings.repository import SettingsRepository
|
||||
|
||||
__all__ = [
|
||||
"PostgresSettingsRepository",
|
||||
"SettingInvalidError",
|
||||
"SettingKey",
|
||||
"SettingNotFoundError",
|
||||
"SettingsDatabaseUnavailableError",
|
||||
"SettingsReader",
|
||||
"SettingsRepository",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,36 @@
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Generic, TypeVar
|
||||
|
||||
T = TypeVar("T")
|
||||
_MISSING = object()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SettingKey(Generic[T]):
|
||||
name: str
|
||||
decoder: Callable[[str], T]
|
||||
required: bool = True
|
||||
default: Any = _MISSING
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name or self.name.strip() != self.name:
|
||||
raise ValueError("setting key name must be non-empty")
|
||||
if self.required and self.default is not _MISSING:
|
||||
raise ValueError("a required setting cannot have a default")
|
||||
|
||||
@property
|
||||
def has_default(self) -> bool:
|
||||
return self.default is not _MISSING
|
||||
|
||||
|
||||
class SettingNotFoundError(LookupError):
|
||||
pass
|
||||
|
||||
|
||||
class SettingInvalidError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class SettingsDatabaseUnavailableError(RuntimeError):
|
||||
pass
|
||||
@@ -0,0 +1,43 @@
|
||||
from collections.abc import Mapping, Sequence
|
||||
import re
|
||||
from typing import Any, AsyncContextManager, Protocol
|
||||
|
||||
from plba.settings.models import SettingsDatabaseUnavailableError
|
||||
|
||||
|
||||
class PostgresConnection(Protocol):
|
||||
async def fetch(self, query: str, *parameters: object) -> Sequence[Mapping[str, Any]]: ...
|
||||
|
||||
|
||||
class PostgresPool(Protocol):
|
||||
def acquire(self) -> AsyncContextManager[PostgresConnection]: ...
|
||||
|
||||
|
||||
class PostgresSettingsRepository:
|
||||
_SCHEMA = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
def __init__(self, pool: PostgresPool, schema: str) -> None:
|
||||
if not self._SCHEMA.fullmatch(schema):
|
||||
raise ValueError("schema must be a PostgreSQL identifier")
|
||||
self._pool = pool
|
||||
self._table = f'"{schema}".settings'
|
||||
|
||||
async def fetch_values(
|
||||
self, setting_key: str, config_key: str | None = None
|
||||
) -> list[str]:
|
||||
query = (
|
||||
f"SELECT setting_value FROM {self._table} "
|
||||
"WHERE setting_key = $1 AND ($2 IS NULL OR config_key = $2) "
|
||||
"AND is_enabled = TRUE ORDER BY sort_order ASC, id ASC"
|
||||
)
|
||||
try:
|
||||
async with self._pool.acquire() as connection:
|
||||
rows = await connection.fetch(query, setting_key, config_key)
|
||||
values = [row["setting_value"] for row in rows]
|
||||
if any(not isinstance(value, str) for value in values):
|
||||
raise TypeError("settings.setting_value must be text")
|
||||
return values
|
||||
except Exception as error:
|
||||
raise SettingsDatabaseUnavailableError(
|
||||
"settings query failed"
|
||||
) from error
|
||||
@@ -0,0 +1,35 @@
|
||||
from typing import TypeVar
|
||||
|
||||
from plba.settings.models import (
|
||||
SettingInvalidError,
|
||||
SettingKey,
|
||||
SettingNotFoundError,
|
||||
)
|
||||
from plba.settings.repository import SettingsRepository
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class SettingsReader:
|
||||
def __init__(self, repository: SettingsRepository) -> None:
|
||||
self._repository = repository
|
||||
|
||||
async def read(
|
||||
self, key: SettingKey[T], config_key: str | None = None
|
||||
) -> list[T]:
|
||||
raw_values = await self._repository.fetch_values(key.name, config_key)
|
||||
if not raw_values:
|
||||
if key.has_default:
|
||||
return [key.default]
|
||||
if key.required:
|
||||
raise SettingNotFoundError(f"required setting not found: {key.name}")
|
||||
return []
|
||||
decoded: list[T] = []
|
||||
for index, value in enumerate(raw_values):
|
||||
try:
|
||||
decoded.append(key.decoder(value))
|
||||
except Exception as error:
|
||||
raise SettingInvalidError(
|
||||
f"setting {key.name} value at index {index} is invalid"
|
||||
) from error
|
||||
return decoded
|
||||
@@ -0,0 +1,7 @@
|
||||
from typing import Protocol
|
||||
|
||||
|
||||
class SettingsRepository(Protocol):
|
||||
async def fetch_values(
|
||||
self, setting_key: str, config_key: str | None = None
|
||||
) -> list[str]: ...
|
||||
@@ -0,0 +1,19 @@
|
||||
from plba.tracing.context import business_trace_context, current_business_trace_id
|
||||
from plba.tracing.models import BusinessTraceContext, BusinessTraceMessage
|
||||
from plba.tracing.postgres import (
|
||||
BusinessTraceContractError,
|
||||
BusinessTraceWriteError,
|
||||
PostgresBusinessTraceWriter,
|
||||
)
|
||||
from plba.tracing.repository import BusinessTraceWriter
|
||||
|
||||
__all__ = [
|
||||
"BusinessTraceContext",
|
||||
"BusinessTraceContractError",
|
||||
"BusinessTraceMessage",
|
||||
"BusinessTraceWriteError",
|
||||
"BusinessTraceWriter",
|
||||
"PostgresBusinessTraceWriter",
|
||||
"business_trace_context",
|
||||
"current_business_trace_id",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,23 @@
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar, Token
|
||||
from collections.abc import Iterator
|
||||
from uuid import UUID
|
||||
|
||||
_BUSINESS_TRACE_ID: ContextVar[str | None] = ContextVar(
|
||||
"plba_business_trace_id", default=None
|
||||
)
|
||||
|
||||
|
||||
def current_business_trace_id() -> str | None:
|
||||
return _BUSINESS_TRACE_ID.get()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def business_trace_context(business_trace_id: UUID | str) -> Iterator[None]:
|
||||
if not str(business_trace_id):
|
||||
raise ValueError("business_trace_id must be non-empty")
|
||||
token: Token[str | None] = _BUSINESS_TRACE_ID.set(str(business_trace_id))
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
_BUSINESS_TRACE_ID.reset(token)
|
||||
@@ -0,0 +1,52 @@
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from types import MappingProxyType
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
|
||||
def _validate_event_time(value: datetime) -> None:
|
||||
if value.tzinfo is None or value.utcoffset() is None:
|
||||
raise ValueError("event_time must include a timezone")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BusinessTraceContext:
|
||||
business_trace_id: UUID
|
||||
alias: str
|
||||
trace_type: str
|
||||
event_time: datetime
|
||||
attributes: Mapping[str, Any] = field(default_factory=dict)
|
||||
parent_id: UUID | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.business_trace_id, UUID):
|
||||
raise ValueError("business_trace_id must be a UUID")
|
||||
if self.parent_id is not None and not isinstance(self.parent_id, UUID):
|
||||
raise ValueError("parent_id must be a UUID or None")
|
||||
if not isinstance(self.alias, str) or not isinstance(self.trace_type, str):
|
||||
raise ValueError("alias and trace_type must be strings")
|
||||
if not self.alias or not self.trace_type:
|
||||
raise ValueError("alias and trace_type must be non-empty")
|
||||
_validate_event_time(self.event_time)
|
||||
object.__setattr__(self, "attributes", MappingProxyType(dict(self.attributes)))
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BusinessTraceMessage:
|
||||
business_trace_id: UUID
|
||||
event_time: datetime
|
||||
step: str
|
||||
status: str
|
||||
level: str
|
||||
message: str
|
||||
attributes: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.business_trace_id, UUID):
|
||||
raise ValueError("business_trace_id must be a UUID")
|
||||
_validate_event_time(self.event_time)
|
||||
if any(not isinstance(value, str) for value in (self.step, self.status, self.level, self.message)):
|
||||
raise ValueError("step, status, level and message must be strings")
|
||||
object.__setattr__(self, "attributes", MappingProxyType(dict(self.attributes)))
|
||||
@@ -0,0 +1,92 @@
|
||||
from typing import AsyncContextManager
|
||||
import json
|
||||
import re
|
||||
from typing import Any, Protocol
|
||||
|
||||
from plba.tracing.models import BusinessTraceContext, BusinessTraceMessage
|
||||
|
||||
|
||||
class BusinessTraceContractError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class BusinessTraceWriteError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class TraceConnection(Protocol):
|
||||
async def execute(self, query: str, *parameters: object) -> str: ...
|
||||
|
||||
|
||||
class TracePool(Protocol):
|
||||
def acquire(self) -> AsyncContextManager[TraceConnection]: ...
|
||||
|
||||
|
||||
class PostgresBusinessTraceWriter:
|
||||
_SCHEMA = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
def __init__(self, pool: TracePool, schema: str, application_id: str) -> None:
|
||||
if not self._SCHEMA.fullmatch(schema):
|
||||
raise ValueError("schema must be a PostgreSQL identifier")
|
||||
if not isinstance(application_id, str) or not application_id:
|
||||
raise ValueError("application_id must be non-empty")
|
||||
self._pool = pool
|
||||
self._application_id = application_id
|
||||
self._contexts = f'"{schema}".trace_contexts'
|
||||
self._messages = f'"{schema}".trace_messages'
|
||||
|
||||
async def write_context(self, context: BusinessTraceContext) -> None:
|
||||
attributes = self._json_attributes(context.attributes)
|
||||
query = (
|
||||
f"INSERT INTO {self._contexts} "
|
||||
"(trace_id, parent_id, alias, trace_type, event_time, attrs_json) "
|
||||
"VALUES ($1, $2, $3, $4, $5, $6::jsonb)"
|
||||
)
|
||||
await self._execute(
|
||||
query,
|
||||
context.business_trace_id,
|
||||
context.parent_id,
|
||||
context.alias,
|
||||
context.trace_type,
|
||||
context.event_time,
|
||||
attributes,
|
||||
)
|
||||
|
||||
async def write_message(self, message: BusinessTraceMessage) -> None:
|
||||
attributes = self._json_attributes(message.attributes)
|
||||
query = (
|
||||
f"INSERT INTO {self._messages} "
|
||||
"(trace_id, event_time, step, status, level, message, attrs_json) "
|
||||
"VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb)"
|
||||
)
|
||||
await self._execute(
|
||||
query,
|
||||
message.business_trace_id,
|
||||
message.event_time,
|
||||
message.step,
|
||||
message.status,
|
||||
message.level,
|
||||
message.message,
|
||||
attributes,
|
||||
)
|
||||
|
||||
async def _execute(self, query: str, *parameters: object) -> None:
|
||||
try:
|
||||
async with self._pool.acquire() as connection:
|
||||
await connection.execute(query, *parameters)
|
||||
except Exception as error:
|
||||
raise BusinessTraceWriteError("business trace write failed") from error
|
||||
|
||||
def _json_attributes(self, attributes: Any) -> str:
|
||||
if not hasattr(attributes, "items"):
|
||||
raise BusinessTraceContractError("attributes must be a JSON object")
|
||||
values = dict(attributes)
|
||||
if any(not isinstance(key, str) for key in values):
|
||||
raise BusinessTraceContractError("attribute keys must be strings")
|
||||
payload = {**values, "application_id": self._application_id}
|
||||
try:
|
||||
return json.dumps(payload, allow_nan=False)
|
||||
except (TypeError, ValueError) as error:
|
||||
raise BusinessTraceContractError(
|
||||
"attributes must contain JSON-safe values"
|
||||
) from error
|
||||
@@ -0,0 +1,9 @@
|
||||
from typing import Protocol
|
||||
|
||||
from plba.tracing.models import BusinessTraceContext, BusinessTraceMessage
|
||||
|
||||
|
||||
class BusinessTraceWriter(Protocol):
|
||||
async def write_context(self, context: BusinessTraceContext) -> None: ...
|
||||
|
||||
async def write_message(self, message: BusinessTraceMessage) -> None: ...
|
||||
@@ -0,0 +1,23 @@
|
||||
from plba.workflow.engine import WorkflowEngine
|
||||
from plba.workflow.failure_recorder import WorkflowFailureRecorder
|
||||
from plba.workflow.models import (
|
||||
StepResult,
|
||||
WorkflowContext,
|
||||
WorkflowDefinition,
|
||||
WorkflowNode,
|
||||
WorkflowRunResult,
|
||||
WorkflowTransitionError,
|
||||
)
|
||||
from plba.workflow.persistence import WorkflowPersistence
|
||||
|
||||
__all__ = [
|
||||
"StepResult",
|
||||
"WorkflowContext",
|
||||
"WorkflowDefinition",
|
||||
"WorkflowEngine",
|
||||
"WorkflowFailureRecorder",
|
||||
"WorkflowNode",
|
||||
"WorkflowPersistence",
|
||||
"WorkflowRunResult",
|
||||
"WorkflowTransitionError",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,131 @@
|
||||
from collections.abc import Callable, Mapping
|
||||
import logging
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from plba.workflow.models import (
|
||||
StepResult,
|
||||
WorkflowContext,
|
||||
WorkflowDefinition,
|
||||
WorkflowRunResult,
|
||||
WorkflowTransitionError,
|
||||
)
|
||||
from plba.workflow.persistence import WorkflowPersistence
|
||||
from plba.workflow.failure_recorder import WorkflowFailureRecorder
|
||||
|
||||
|
||||
class WorkflowEngine:
|
||||
def __init__(
|
||||
self,
|
||||
persistence: WorkflowPersistence,
|
||||
id_factory: Callable[[], UUID] = uuid4,
|
||||
logger: logging.Logger | None = None,
|
||||
) -> None:
|
||||
self._persistence = persistence
|
||||
self._id_factory = id_factory
|
||||
self._logger = logger or logging.getLogger(__name__)
|
||||
self._failure_recorder = WorkflowFailureRecorder(
|
||||
persistence, self._logger
|
||||
)
|
||||
|
||||
async def run(
|
||||
self, definition: WorkflowDefinition, context: WorkflowContext
|
||||
) -> WorkflowRunResult:
|
||||
definition.validate()
|
||||
run_id = str(self._id_factory())
|
||||
current_name = definition.initial_step
|
||||
current_context = context
|
||||
await self._persistence.record_run_started(
|
||||
run_id, definition.name, self._snapshot(current_context)
|
||||
)
|
||||
self._logger.info(
|
||||
"workflow run started",
|
||||
extra={
|
||||
"attributes": {"workflow": definition.name, "run_id": run_id}
|
||||
},
|
||||
)
|
||||
try:
|
||||
while current_name is not None:
|
||||
node = definition.nodes[current_name]
|
||||
self._logger.debug(
|
||||
"workflow step started",
|
||||
extra={
|
||||
"attributes": {
|
||||
"workflow": definition.name,
|
||||
"run_id": run_id,
|
||||
"step": current_name,
|
||||
}
|
||||
},
|
||||
)
|
||||
await self._persistence.record_step_started(
|
||||
run_id, current_name, self._snapshot(current_context)
|
||||
)
|
||||
try:
|
||||
result = await node.step(current_context)
|
||||
next_name = self._next_node(node.transitions, result)
|
||||
except Exception as error:
|
||||
await self._failure_recorder.record_step_failure(
|
||||
run_id, current_name, current_context, error
|
||||
)
|
||||
raise
|
||||
|
||||
next_context = current_context.with_updates(result.updates)
|
||||
await self._persistence.record_step_completed(
|
||||
run_id, current_name, self._snapshot(next_context), result
|
||||
)
|
||||
self._logger.debug(
|
||||
"workflow step completed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"workflow": definition.name,
|
||||
"run_id": run_id,
|
||||
"step": current_name,
|
||||
"status": result.status,
|
||||
}
|
||||
},
|
||||
)
|
||||
current_name = next_name
|
||||
current_context = next_context
|
||||
await self._persistence.record_run_completed(
|
||||
run_id, definition.name, self._snapshot(current_context), "completed"
|
||||
)
|
||||
self._logger.info(
|
||||
"workflow run completed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"workflow": definition.name,
|
||||
"run_id": run_id,
|
||||
"status": "completed",
|
||||
}
|
||||
},
|
||||
)
|
||||
return WorkflowRunResult(run_id, current_context, "completed")
|
||||
except Exception as error:
|
||||
await self._failure_recorder.record_run_failure(
|
||||
run_id, definition.name, self._snapshot(current_context), error
|
||||
)
|
||||
raise
|
||||
|
||||
@staticmethod
|
||||
def _next_node(
|
||||
transitions: Mapping[str, str], result: StepResult
|
||||
) -> str | None:
|
||||
if not isinstance(result, StepResult):
|
||||
raise TypeError("workflow step must return StepResult")
|
||||
if not isinstance(result.updates, Mapping) or not isinstance(result.status, str):
|
||||
raise TypeError("workflow step returned an invalid StepResult")
|
||||
if result.transition is None:
|
||||
if transitions:
|
||||
raise WorkflowTransitionError(
|
||||
"step must select one of its declared transitions"
|
||||
)
|
||||
return None
|
||||
if result.transition not in transitions:
|
||||
raise WorkflowTransitionError(
|
||||
f"transition is not declared: {result.transition}"
|
||||
)
|
||||
return transitions[result.transition]
|
||||
|
||||
@staticmethod
|
||||
def _snapshot(context: WorkflowContext) -> WorkflowContext:
|
||||
return WorkflowContext(context.state)
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
import logging
|
||||
|
||||
from plba.workflow.models import WorkflowContext
|
||||
from plba.workflow.persistence import WorkflowPersistence
|
||||
|
||||
|
||||
class WorkflowFailureRecorder:
|
||||
def __init__(
|
||||
self, persistence: WorkflowPersistence, logger: logging.Logger
|
||||
) -> None:
|
||||
self._persistence = persistence
|
||||
self._logger = logger
|
||||
|
||||
async def record_step_failure(
|
||||
self,
|
||||
run_id: str,
|
||||
step_name: str,
|
||||
context: WorkflowContext,
|
||||
error: Exception,
|
||||
) -> None:
|
||||
try:
|
||||
await self._persistence.record_step_failed(
|
||||
run_id, step_name, self._snapshot(context), error
|
||||
)
|
||||
except Exception as persistence_error:
|
||||
error.add_note(f"could not persist workflow step failure: {persistence_error}")
|
||||
self._logger.error(
|
||||
"workflow step failed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"run_id": run_id,
|
||||
"step": step_name,
|
||||
"error_type": type(error).__name__,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
async def record_run_failure(
|
||||
self,
|
||||
run_id: str,
|
||||
workflow_name: str,
|
||||
context: WorkflowContext,
|
||||
error: Exception,
|
||||
) -> None:
|
||||
try:
|
||||
await self._persistence.record_run_failed(
|
||||
run_id, workflow_name, self._snapshot(context), error
|
||||
)
|
||||
except Exception as persistence_error:
|
||||
error.add_note(f"could not persist workflow failure: {persistence_error}")
|
||||
self._logger.error(
|
||||
"workflow run failed",
|
||||
extra={
|
||||
"attributes": {
|
||||
"workflow": workflow_name,
|
||||
"run_id": run_id,
|
||||
"error_type": type(error).__name__,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _snapshot(context: WorkflowContext) -> WorkflowContext:
|
||||
return WorkflowContext(context.state)
|
||||
@@ -0,0 +1,87 @@
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from copy import deepcopy
|
||||
from types import MappingProxyType
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkflowContext:
|
||||
state: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
state = deepcopy(dict(self.state))
|
||||
if any(not isinstance(key, str) for key in state):
|
||||
raise ValueError("workflow context keys must be strings")
|
||||
object.__setattr__(self, "state", MappingProxyType(state))
|
||||
|
||||
def with_updates(self, updates: Mapping[str, Any]) -> "WorkflowContext":
|
||||
return WorkflowContext({**self.state, **updates})
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class StepResult:
|
||||
transition: str | None = None
|
||||
updates: Mapping[str, Any] = field(default_factory=dict)
|
||||
status: str = "completed"
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "updates", MappingProxyType(deepcopy(dict(self.updates)))
|
||||
)
|
||||
if not isinstance(self.status, str) or not self.status:
|
||||
raise ValueError("step status must be a non-empty string")
|
||||
if self.transition is not None and not isinstance(self.transition, str):
|
||||
raise ValueError("step transition must be a string or None")
|
||||
if any(not isinstance(key, str) for key in self.updates):
|
||||
raise ValueError("step update keys must be strings")
|
||||
|
||||
|
||||
WorkflowStep = Callable[[WorkflowContext], Awaitable[StepResult]]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkflowNode:
|
||||
step: WorkflowStep
|
||||
transitions: Mapping[str, str] = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "transitions", MappingProxyType(dict(self.transitions)))
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkflowDefinition:
|
||||
name: str
|
||||
initial_step: str
|
||||
nodes: Mapping[str, WorkflowNode]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "nodes", MappingProxyType(dict(self.nodes)))
|
||||
self.validate()
|
||||
|
||||
def validate(self) -> None:
|
||||
if not self.name or self.name.strip() != self.name:
|
||||
raise ValueError("workflow name must be non-empty")
|
||||
if not self.nodes or self.initial_step not in self.nodes:
|
||||
raise ValueError("workflow initial_step must name a declared node")
|
||||
for node_name, node in self.nodes.items():
|
||||
if not node_name or node_name.strip() != node_name:
|
||||
raise ValueError("workflow node names must be non-empty")
|
||||
for transition, target in node.transitions.items():
|
||||
if (
|
||||
not isinstance(transition, str)
|
||||
or not transition
|
||||
or not isinstance(target, str)
|
||||
or target not in self.nodes
|
||||
):
|
||||
raise ValueError(f"invalid transition declared by node {node_name}")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkflowRunResult:
|
||||
run_id: str
|
||||
context: WorkflowContext
|
||||
status: str
|
||||
|
||||
|
||||
class WorkflowTransitionError(ValueError):
|
||||
pass
|
||||
@@ -0,0 +1,45 @@
|
||||
from typing import Protocol
|
||||
|
||||
from plba.workflow.models import StepResult, WorkflowContext
|
||||
|
||||
|
||||
class WorkflowPersistence(Protocol):
|
||||
async def record_run_started(
|
||||
self, run_id: str, workflow_name: str, context: WorkflowContext
|
||||
) -> None: ...
|
||||
|
||||
async def record_step_started(
|
||||
self, run_id: str, step_name: str, context: WorkflowContext
|
||||
) -> None: ...
|
||||
|
||||
async def record_step_completed(
|
||||
self,
|
||||
run_id: str,
|
||||
step_name: str,
|
||||
context: WorkflowContext,
|
||||
result: StepResult,
|
||||
) -> None: ...
|
||||
|
||||
async def record_step_failed(
|
||||
self,
|
||||
run_id: str,
|
||||
step_name: str,
|
||||
context: WorkflowContext,
|
||||
error: BaseException,
|
||||
) -> None: ...
|
||||
|
||||
async def record_run_completed(
|
||||
self,
|
||||
run_id: str,
|
||||
workflow_name: str,
|
||||
context: WorkflowContext,
|
||||
status: str,
|
||||
) -> None: ...
|
||||
|
||||
async def record_run_failed(
|
||||
self,
|
||||
run_id: str,
|
||||
workflow_name: str,
|
||||
context: WorkflowContext,
|
||||
error: BaseException,
|
||||
) -> None: ...
|
||||
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user