04 / Failure, by design
Find out how your client behaves when an API becomes slow or unreliable
Inject latency, errors, timeouts, and intermittent failures into controlled API traffic so resilience decisions are based on observed behavior.
+1200msretry observedfallback: yesWorking reference
What goes in, what comes out, and what the result can prove.
Concrete artifact
Exercise a retry budget without breaking production
target: https://staging-api.example/orders
forceStatus: 503
errorRate: 0.20
latencyMs: 800
maxExecutions: 25Review sequence
Three checks before the result becomes a decision.
Bound the blast radius
Set target, expiry, execution count and rate limit.
Observe the client
Capture attempts, backoff, timeout and final user-visible result.
Remove the rule
Disable the experiment and verify normal behavior returns.
Limits
What this workflow does not prove.
- Run only against systems you own or are authorized to test.
- Injected failure validates client behavior, not the root cause of a real incident.
- Production experiments require independent safeguards and an abort path.
Questions worth answering
Before you put it into a real workflow.
Does this modify the real API?
No. Failure behavior is introduced through a controlled proxy path rather than changing the upstream implementation.
What failures can I model?
Common experiments include latency, HTTP errors, timeouts, connection failures, and probabilistic behavior.
Should I point production traffic at it?
Start in development or staging with bounded traffic. Production experiments require explicit ownership, safeguards, and rollback planning.
Next move
Start with one concrete API problem.
Keep the first step small. Move into a workspace when the result deserves to be saved, repeated, or shared.