Stage 12: Microservices, lesson 4 of 7

Resilience: timeouts, retries and circuit breakers

Advanced3 min read@since 17Code runs on your Java 25
Explain it forThe essentials plus production detail and pitfalls.

In distributed systems, failure is normal. Protect every remote call:

  • Timeouts: never wait forever. Set connect and read timeouts on every client.
  • Retries with exponential backoff and jitter, for temporary errors only, and only for idempotent operations.
  • Circuit breaker: after too many failures it opens and fails fast; after a wait it goes half-open to test; success closes it again.
  • Bulkhead: limit concurrent calls so one slow dependency can't use up every thread.
  • Rate limiters and fallbacks (a cached or default response).

The usual tool is Resilience4j (Netflix Hystrix is retired). Spring Framework 7 adds built-in @Retryable and @ConcurrencyLimit, so spring-retry is no longer needed for simple cases.

Example

Java
@Service
public class RecommendationClient {
    private final RestClient http;

    RecommendationClient(RestClient.Builder builder) {
        this.http = builder.baseUrl("http://recommendation-service").build();
    }

    @CircuitBreaker(name = "recs", fallbackMethod = "popular")
    @Retry(name = "recs")
    public List<CourseCard> forUser(String userId) {
        return http.get().uri("/users/{id}/recs", userId)
                   .retrieve()
                   .body(new ParameterizedTypeReference<List<CourseCard>>() {});
    }

    private List<CourseCard> popular(String userId, Throwable t) {
        return cache.popularCourses();           // graceful fallback
    }
}
application.yml
resilience4j:
  circuitbreaker:
    instances:
      recs:
        sliding-window-size: 20
        failure-rate-threshold: 50
        wait-duration-in-open-state: 30s
  retry:
    instances:
      recs:
        max-attempts: 3
        wait-duration: 200ms
        enable-exponential-backoff: true

Common mistake

Retrying non-idempotent operations such as "charge card" without an idempotency key. A timeout doesn't mean the charge failed; retrying may charge the customer twice.

Under the hood

Retries multiply load: three retries at each of three layers can mean up to 27 calls to the bottom service during an outage (a retry storm). Retry at one layer only, with backoff and a limit. In Resilience4j's default order the Retry wraps the CircuitBreaker, so every retry attempt is counted by the breaker.

Check yourself

What does an open circuit breaker do?

How this connects

Where this leads

You've reached the end of this thread. Try a learning path for what's next.

Part of Microservices and production.

Was this lesson helpful?

Finished reading? Mark it complete to track your progress.