Third-Party Integration Resilience

Third-party integration resilience means keeping your own product understandable when a supplier is slow, unavailable, or uncertain about what happened. Start by defining what the user can safely do during failure, then design how the two systems converge afterward. A retry loop covers only a small part of that job.

A useful boundary has three answers: how long you will wait, whether repeating an operation is safe, and how you will discover its authoritative outcome. If any answer is missing, document an unresolved state instead of guessing success or failure. Your interface should be more honest than the network.

TL;DR

Quick Example

This Python example models a fictional provider that creates an object and then loses its response. Its dictionary lookup stands in for an authoritative receipt endpoint; it is not a real integration or durable idempotency service.

The model also rejects reusing a key with different input. Test concurrent calls, process restarts, retention, and the provider's real guarantees separately. In a deployed integration, the lookup may itself be delayed or unavailable. That case remains pending rather than becoming permission to create a second operation.

Core Concepts

AWS explains that timeouts bound resource use, while retries can increase load on an already failing dependency. Side effects may already have happened when a timeout reaches the caller. Backoff and jitter help control retry pressure; they do not resolve uncertain business outcomes. AWS Builders' Library.

Separate transport state from operation state. “Connection failed” describes observation. “Order rejected” describes business outcome. Between them sits “outcome unknown,” which needs a durable record, correlation identifier, next check, and responsible owner.

Idempotency is a provider-specific contract. Stripe's API v1, for example, records results after execution starts, checks repeated parameters, and has retention conditions. Do not assume another supplier implements those semantics merely because it accepts a similarly named header. Stripe idempotent requests.

Choose a failure policy by consequence

Use the following matrix as an editorial design aid. Its rows describe proposed operating decisions, not measured supplier behavior.

An optional image preview and an inventory reservation should not share a default failure policy. Serving stale previews may be tolerable; inventing available inventory may create an obligation the system cannot fulfill. Have the product owner choose acceptable degraded behavior before the integration fails.

Define freshness at the point of use. A cache can contain a technically valid object that is too old for the action being offered. Include the observation timestamp and the scope covered, and avoid presenting a partial response as a complete empty result.

Bound work before adding retries

Allocate one end-to-end deadline. Reserve time for your own processing and the user's response, then divide the remaining budget among attempts. Account for connection setup and response reading in the actual client. A nominal per-attempt timeout multiplied by several retry layers can exceed the request's useful lifetime.

Choose one retry owner for each boundary. If an SDK retries, a worker retries, and the queue redelivers, a single logical operation can generate many attempts. Record logical operation ID separately from attempt ID so dashboards reveal the multiplication.

For a circuit breaker, describe what opens it, how long the decision remains valid, which limited probes are allowed, and how recovery is judged. A breaker is a local protection mechanism, not proof that the provider is down everywhere. Avoid allowing half-open probes to become an uncontrolled second workload.

Use admission limits and queue limits together. An unbounded queue can preserve every request while making each one uselessly late. Specify when to reject, expire, compensate, or request human review. Preserve the reason in a status that support can explain.

Reconciliation is part of the integration

Persist the operation's intent before relying on a later background check. Capture a stable operation identifier, normalized request identity, creation time, last observation, and next reconciliation time. Avoid retaining secrets or complete sensitive payloads merely to simplify debugging.

Match the reconciliation method to the supplier. Options might include a receipt lookup, an export joined by an external reference, or a documented replay contract. A webhook alone is insufficient if events can be delayed or missed; a periodic comparison may be necessary. Design the comparison around authoritative identifiers and explicit conflict rules.

Suppose a fictional catalog request asks for ten item IDs and returns nine valid objects with a successful HTTP status. Check the requested and returned ID sets. An absent object could mean deletion, lack of permission, pagination, or supplier failure. Treating absence as deletion without a documented contract turns a transient outage into permanent local damage.

Recovery should also be observable. Count unresolved operations by age and consequence, record the oldest pending item, and sample disagreement cases. A falling error rate can coexist with a growing pile of uncertain writes. Close the incident when that uncertainty has been resolved or explicitly accepted, not merely when requests become green again.

Best Practices

Build a contract test suite around semantic outcomes: accepted, rejected, unknown, partial, stale, and conflicting. Add a response-lost-after-commit case and a process-restart case. These test different obligations and should have different expected results.

Document sandbox gaps. A supplier's test environment may not reproduce production quotas, delayed events, authentication expiry, or partial failures. Maintain a gap list and arrange a bounded, authorized rollout with observation and rollback criteria. Do not turn sandbox success into a reliability claim.

Keep a supplier runbook containing support contact procedure, documented limits, credentials ownership, safe degradation, and reconciliation ownership. Store sensitive access material in the appropriate secret system rather than the runbook itself. Revisit the contract when the provider changes its API or retention rules.

Common Mistakes

Bad: Generate a new idempotency key after every timeout.

Correct: Preserve the logical operation identity and follow the supplier's replay or lookup rules. A genuinely new operation needs a new identity; an uncertain old one needs reconciliation.

Bad: Declare recovery when the provider returns successful status codes.

Correct: Check pending writes, missing records, stale caches, and backlog age before declaring business recovery.

Bad: Retry every error with the same schedule.

Correct: Distinguish validation and authorization failures from transient conditions, honor documented limits, and stop attempts when the operation budget expires.

FAQ

Is a circuit breaker enough?

No. It can limit calls during suspected failure, but it does not determine whether an earlier write committed or whether returned data is complete. Pair it with operation tracking and recovery rules.

Should every supplier call be asynchronous?

Use asynchronous work when the user can meaningfully continue with a pending state. If an immediate answer is required, define a bounded synchronous attempt and an honest failure path. Moving work to a queue does not remove the business deadline.

What would reverse a caching recommendation?

A requirement for current authorization, inventory, or another consequential fact can make stale data unacceptable. Test the actual decision made from the cached object rather than treating all reads as harmless.

Related Topics

References