Skip to content

Observability

Observability is the authority for what the system is doing now. It closes the loop from generated intent to production evidence without automatically rewriting the model.

Every runtime signal should be attributable to:

  • project and application;
  • model revision;
  • generator version;
  • release identifier;
  • environment and machine;
  • request, job, or event correlation ID.

Without this chain, a shared generator fix cannot reliably identify which releases are affected.

  • build, migration, readiness, smoke, and promotion duration;
  • active and previous release;
  • candidate failures and retry count;
  • managed-state versus provider-state divergence;
  • time since last successful regeneration.
  • request rate, error rate, and latency;
  • saturation, memory, CPU, connections, and disk;
  • queue depth, age, retry count, and dead letters;
  • cache hit rate and eviction;
  • database lock time, slow queries, and migration state.
  • domain error categories and rejected actions;
  • lifecycle transitions and stuck states;
  • provider latency, failures, and circuit state;
  • callback signature failures and replay detection;
  • business outcomes the model identifies as important.

Use separate endpoints or semantics:

  • liveness: process is responsive;
  • readiness: migrations and required dependencies allow traffic;
  • integration status: optional and external capabilities are configured and reachable;
  • version: release, revision, and generator identity.

Do not run expensive external calls on every liveness probe.

Alert on user impact or exhaustion risk, not every log line. Good alerts identify the failing layer and link to a runbook.

Examples:

  • readiness remains false after candidate startup budget;
  • error-rate or latency SLO burns rapidly after promotion;
  • migration exceeds lock/runtime limits;
  • oldest queue item exceeds the product SLA;
  • provider circuit remains open;
  • published app volume is missing or nearly full;
  • managed files or release state diverge.

Telemetry can reveal a missing workload constraint, rule, or integration outcome. A human then decides whether to revise the model, improve the generator, or operate infrastructure. Automatic mutation would confuse observed reality with desired behavior.

See Incidents and Troubleshooting.