Incidents and recovery
An incident is not automatically a code change. The first response is to classify which authority is failing, restore a safe service state, and then make the permanent correction in the owning layer.
Incident classifier
Section titled “Incident classifier”| Scenario | Immediate response | Permanent response |
|---|---|---|
| Candidate fails during deployment | Keep active release; stop promotion | Fix model, generator, config, or infra cause |
| New release regresses | Traffic rollback if compatible | New revision or generator fix |
| Third-party API unavailable | Retry/circuit/queue/degrade | Provider policy or adapter improvement |
| Third-party API contract changed | Disable affected path or use compatible config | Update integration contract/adapter and release |
| Credential or endpoint invalid | Correct control-plane configuration | Improve validation/rotation process |
| General load spike | Scale, rate-limit, shed load | Capacity policy |
| Query degrades at high volume | Protect service, limit workload, scale read path | Generator strategy or modeled workload requirement |
| Queue backlog | Add workers, pause producers, replay safely | Topology and backpressure policy |
| Corrupt data | Quarantine/restore/repair | Forward repair and invariant improvement |
| Generator defect | Roll back affected release | Fix shared generator and regenerate |
| Desired behavior is wrong | Use safe feature/config controls if declared | Revise model and release |
Third-party failures
Section titled “Third-party failures”The product can remain correct while a dependency is unavailable. Generated adapters should support bounded timeouts, retry policy, circuit breaking, idempotency, and a visible pending or degraded state where appropriate.
Changing an add-on endpoint or key is a configuration operation. Changing the contract or desired failure behavior is a model/generator change.
Scalability
Section titled “Scalability”Distinguish capacity from algorithmic shape:
- capacity problem: the implementation behaves correctly and needs more instances, connections, workers, or cache;
- strategy problem: the generated query, algorithm, or data model does not scale at the real workload.
Infrastructure handles the first. The second requires a generator improvement or an explicit workload decision in the model. “Infra will scale it” cannot repair an unbounded scan or an incompatible online migration.
Business urgency
Section titled “Business urgency”An urgent permission or business-rule change still belongs in the model. The platform’s job is to make model → validate → generate → verify → promote fast enough for incident use.
If the needed behavior cannot be expressed, the honest choices are:
- extend the model/generator;
- activate a predeclared configuration or feature policy;
- export and explicitly fork the application;
- keep the unsafe behavior disabled.
Silently editing a managed file creates a fix the next regeneration can erase.
Runbook sequence
Section titled “Runbook sequence”- Detect and declare impact.
- Identify active release, schema state, configuration revision, and provider state.
- Classify the owning authority.
- Stabilize through traffic, capacity, config, queue, repair, or compatible rollback.
- Create the permanent model/generator/operational correction.
- Generate and verify a new candidate.
- Promote and observe.
- Record evidence and improve the shared guardrail.
See Observability and Troubleshooting.