What is the difference between rate limiting and load shedding?

Rate limiting restricts individual clients to a maximum number of requests per time window (e.g., 100 requests/minute per API key), protecting against abuse regardless of server capacity. Load shedding rejects requests based on the server's current capacity — when the server is overloaded, it drops requests to stay responsive. Rate limiting is per-client and always active; load shedding is system-wide and only activates under pressure. Use both: rate limiting to prevent abuse, load shedding to survive genuine traffic spikes.

When should I return a 429 vs a 503 status code for throttled requests?

Return 429 Too Many Requests when a specific client has exceeded their rate limit — the server is fine, but this particular client is sending too much. Return 503 Service Unavailable when the server itself is overloaded and cannot process requests from any client. Both should include a Retry-After header. The distinction matters for client behavior: 429 tells the client to slow down its own requests, while 503 tells the client the entire service needs time to recover.

How does the circuit breaker pattern prevent cascading failures?

When service A calls service B and B becomes slow or unresponsive, A's threads block waiting for B, eventually exhausting A's capacity and causing A to fail too — a cascade. A circuit breaker tracks B's failure rate. When failures exceed a threshold, the breaker 'opens' and immediately returns an error without calling B, preventing thread exhaustion in A. After a timeout period, the breaker enters 'half-open' state and sends a test request. If B responds successfully, the breaker closes and normal traffic resumes.

What auto-scaling metric should I use to handle traffic spikes effectively?

Use ALBRequestCountPerTarget (requests per instance) as your primary scaling metric rather than CPU. CPU-based scaling reacts too slowly — by the time CPU spikes, requests are already queuing. Request-count scaling anticipates capacity needs based on traffic volume. Set scale-out cooldown to 60 seconds for fast reaction and scale-in cooldown to 300 seconds to prevent flapping. Combine with scheduled scaling for predictable traffic patterns (e.g., business hours) and step scaling for sudden spikes.

What is the bulkhead pattern and when should I use it?

The bulkhead pattern isolates different workload types into separate resource pools (connection pools, thread pools, concurrency limits). Without bulkheads, a spike in one type of work (e.g., report generation) can exhaust all available connections and starve critical operations (e.g., API reads). With bulkheads, each workload type has a fixed allocation — reports get max 10 concurrent slots, API reads get 100, webhooks get 20. When one bulkhead fills up, the others are unaffected. Use it whenever your service handles workloads with different performance characteristics.

Server Overload and Throttling