Mirror the catalog and keep it true
Holding a copy is allowed and often right. This is the order to build it in: backfill, deltas, revalidation, and a sweep for what the first three missed.
Decide whether you need a copy at all
A mirror is a second database to keep honest. Take one only when you need something the API cannot do for you.
- You need it if you join our data to your own (stock, prices, ownership) or query it in ways no endpoint expresses.
- You do not need it for speed alone. Records are cacheable for hours and a repeat read costs nothing against your quota; see caching first.
- You do not need it to survive an outage either. A cache of the pages you actually serve does that with a fraction of the work.
The order that does not lose rows
Each step covers a gap the step before it leaves open. Skipping one is the reason a mirror drifts.
- Backfill onceWalk the catalog in creation order and store what you get, with the cursor you stopped at.
- Subscribe before you finishRegister the webhook endpoint before the walk ends, so a record changed mid-walk is not lost between the two.
- Take deltas from the webhookCreated, updated and deleted events arrive within about fifteen minutes of the change and carry the record.
- Revalidate on a scheduleSweep what you hold with the ETag you stored. Unchanged records answer 304 and cost nothing.
- Reconcile periodicallyWalk the catalog again on a slow cycle to catch anything missed while your endpoint was down.
The first walk
Sort by creation, not by name. A record added mid-walk then lands at the end instead of shifting a page you have already read. Page depth is capped per plan, and studio, enterprise have no ceiling; the pagination page covers the limits.
// Walk the catalog once, in creation order, and remember where// you stopped. Ordering by `created_at` means a record added// mid-walk lands at the end rather than shifting a page you have// already read. A name-ordered walk silently skips rows.let page = 1let newest = loadCursor() // the created_at you last stored, or null for (;;) { const url = new URL('https://api.gunspec.io/v1/firearms') url.searchParams.set('sort', 'created_at') url.searchParams.set('order', 'asc') url.searchParams.set('per_page', '100') url.searchParams.set('page', String(page)) // Only what is new since the last run. On a first run, omitted. if (newest) url.searchParams.set('created_after', newest) const res = await fetch(url, { headers: { 'X-API-Key': key } }) const { data, pagination } = await res.json() if (data.length === 0) break await upsertAll(data) newest = data[data.length - 1].createdAt saveCursor(newest) // A short page is the last page. `pagination.totalPages` is not // sent to every plan, so ending on it works on some keys and // loops forever on others. if (data.length < pagination.per_page) break page++}Deltas arrive as webhooks
The database itself records what changed and a job drains that record every quarter hour, so a change made by an import you never see still reaches you. The webhooks page has the full event list and the delivery rules.
| Event | What it means for your copy |
|---|---|
firearm.created | A record you do not hold. Insert it; the event carries the record as the API would return it. |
firearm.updated | A field we serve changed. Replace your copy and store the new version. |
firearm.deleted | The record is gone. The payload is the id alone, which is all a deletion has to say. |
firearm.confidence.changed | The record is the same but how well it is sourced is not. Worth re-rendering anything that shows confidence. |
image.created | New media for a record you may already hold. Refresh its media list rather than the record. |
// Your endpoint. Verify first, answer 2xx fast, work afterwards.import { verifyWebhookSignature } from '@gunspec/sdk' export async function POST(request) { const raw = await request.text() // the exact bytes, not a re-serialised object try { await verifyWebhookSignature(raw, request.headers.get('X-Webhook-Signature'), secret) } catch { // Unsigned or stale. Never act on it, and do not 200 it either. return new Response('bad signature', { status: 401 }) } const event = JSON.parse(raw) // Deliveries retry, so the same event can arrive twice. Key the // work on the event id and make the second one a no-op. await enqueueOnce(event.id, event) // Acknowledge now; fetch the record on your own time. A handler // that fetches before answering is a handler that times out and // gets redelivered. return new Response(null, { status: 204 })}Revalidation is the cheap half
Send the ETag you stored as If-None-Match. An unchanged record answers 304 with no body, and a 304 is not counted against your daily cap, so sweeping a whole mirror costs a fraction of re-downloading one. Caching and freshness explains which value to store for what.
// A record you hold, checked without downloading it again.const res = await fetch(`${API}/v1/firearms/${row.id}`, { headers: { 'X-API-Key': key, ...(row.etag ? { 'If-None-Match': row.etag } : {}) },}) // Unchanged. No body crossed the wire and a 304 is not counted// against your daily cap, so a nightly sweep of the whole mirror// costs a fraction of one re-download.if (res.status === 304) return row const { data } = await res.json()// Store `version` beside the record: it is a hash of the record's// own fields, so it survives a plan change and a response-shape// change that would both move the ETag.return { ...data, etag: res.headers.get('ETag'), version: data.version }Reconcile, because deliveries fail
Your endpoint will be down at some point, and retries do not last forever. A slow full sweep is what turns that from data loss into a delay.
- Dedupe on the event id, not the delivery id: a retry re-sends the same event and the delivery id is different every time.
- Answer
2xxbefore you do the work. A handler that fetches the record first is a handler that times out and gets redelivered. - Remove anything the catalog no longer serves on the sweep rather than hiding it. A record kept after its deletion event is the one that ends up on a page.