Practice

Handle failure on purpose

Some failures are worth sending again and some can only fail again. Telling them apart is the difference between a slow minute and a wasted quota.

What is worth retrying

Branch on error.reason, never on the message: reasons are stable and never renamed, messages are written for people. The errors page lists every one of them.

Reason	Status	How long to wait
`RATE_LIMITED`	429	Retry-After says 60s. Honour the header, not the number.
`PAGINATION_BURST`	429	Exponential backoff with jitter.
`DAILY_CAP_EXCEEDED`	429	Until the reset: midnight UTC for the daily limit, the 1st for the monthly one. `Retry-After` is the seconds left and `error.resetsAt` the instant. Wait only when it is seconds away, a call refused just before a reset. Otherwise stop, schedule the work for after it, and do not loop.
`MONTHLY_CAP_EXCEEDED`	429	Until the reset: midnight UTC for the daily limit, the 1st for the monthly one. `Retry-After` is the seconds left and `error.resetsAt` the instant. Wait only when it is seconds away, a call refused just before a reset. Otherwise stop, schedule the work for after it, and do not loop.
`DEPENDENCY_UNAVAILABLE`	503	Exponential backoff with jitter.
`MAINTENANCE`	503	Exponential backoff with jitter.
`INTERNAL_ERROR`	500	Exponential backoff with jitter.

What cannot succeed on a retry

These are answers, not outages. Retrying them spends requests to receive the same sentence, and in a loop it is how a key reaches its daily cap by lunchtime.

Reason	Status	What actually fixes it
`KEY_INVALID`	401	Check the key for a typo, or create a new one in your account.
`KEY_DISABLED`	401	Re-enable it in your account, or use another key.
`KEY_EXPIRED`	401	Create a new key in your account.
`PLAN_REQUIRED`	403	Upgrade the plan. A new key on the same plan will get the same answer.
`ACCOUNT_SUSPENDED`	403	Contact support. Rotating the key will not help.
`PAGINATION_DEPTH_EXCEEDED`	403	Narrow the list with filters, or upgrade for deeper paging.
`INVALID_PARAMETER`	400	Correct the named fields and send the request again.
`RESOURCE_NOT_FOUND`	404	Check the id. The list and search endpoints return valid ones.

401 is about the credential, 403 about what it is allowed to do. Reissuing a key fixes the first and changes nothing about the second. Authentication draws the line in full.

A retry loop that behaves

Three rules: only retryable reasons, the server’s own Retry-After when it sends one, and jitter so a fleet that failed together does not retry together. A spent daily or monthly allowance is the exception: wait only when its reset is seconds away, otherwise stop.

1// Retry the failures that can succeed later, and only those.2const RETRYABLE = new Set(['RATE_LIMITED', 'PAGINATION_BURST', 'DAILY_CAP_EXCEEDED', 'MONTHLY_CAP_EXCEEDED', 'DEPENDENCY_UNAVAILABLE', 'MAINTENANCE', 'INTERNAL_ERROR'])3 4// A spent daily or monthly allowance resets at midnight UTC or on the 1st.5// Wait for it only when it is seconds away, as the SDKs do (30 seconds by6// default). Never sleep through a day or a month.7const ALLOWANCES = new Set(['DAILY_CAP_EXCEEDED', 'MONTHLY_CAP_EXCEEDED'])8const MAX_WAIT_SECONDS = 309 10async function call(request, attempt = 0) {11  const res = await fetch(request)12  if (res.ok) return res13 14  const body = await res.json().catch(() => null)15  const reason = body?.error?.reason16 17  // A reason the API says is permanent. Sending it again is a18  // request spent to receive the same sentence.19  if (!RETRYABLE.has(reason) || attempt >= 4) throw new GunSpecError(res.status, body)20 21  // Honour the server's own number when it sends one: it knows when22  // the window resets and a guess does not.23  const retryAfter = Number(res.headers.get('Retry-After')) || 024 25  // The allowance is spent and the reset is far off. Nothing is served26  // before it, so sleeping here is a stuck request, not a retry. Stop, and27  // run the work again after the instant the API names.28  if (ALLOWANCES.has(reason) && retryAfter > MAX_WAIT_SECONDS) {29    throw new LimitReached(reason, body.error.resetsAt)30  }31 32  const backoff = retryAfter * 1000 || 2 ** attempt * 50033 34  // Jitter, or every client that failed together retries together.35  await sleep(backoff + Math.random() * 250)36  return call(request, attempt + 1)37}

The official SDKs do this already: maxRetries defaults to 2, Retry-After is honoured up to maxRetryAfterMs (a refusal whose reset is further off than that is raised at once, never slept through), and every error carries reason and requestId. Tune it with retry on the client rather than wrapping it. See the SDK reference.

What the reader sees meanwhile

A page that renders without our data is better than a page that does not render. Decide this once, at the boundary, rather than in every component.

1// What the reader sees while this is going wrong.2try {3  return await gunspec.firearms.get(id)4} catch (err) {5  // A copy from ten minutes ago is a better page than an error, and6  // the catalog changes on the order of days.7  const stale = await cache.get(id)8  if (stale) return { ...stale, stale: true }9 10  // Nothing cached. Say what failed, quote the request id, and let11  // the page render the rest of itself.12  logger.warn({ requestId: err.requestId, reason: err.reason }, 'gunspec unavailable')13  return null14}

Serve a stale copy before you serve an error. The catalog moves on the order of days, so ten-minute-old specs are not a lie.
Fail one section, not the page: sellers being unavailable should not take the specification table with it.
Put a timeout on every call. A request that never returns is worse than one that fails, because nothing downstream ever gets to decide.
Never retry a write because the response was slow unless you have an idempotency strategy. A duplicate is a real cost; a delay is not.

When it is us

Every response carries X-Request-Id, and every SDK error carries it as requestId. Log it. With that id we can find the exact request; without it, a report is a description of a page. Support and feedback is where it goes.