Syncing a catalogue

How to keep a store in step with a feed that changes without telling you.

Paging

Search is page-numbered. Ask for pageNumber 1, 2, 3… with a pageSize of up to 50, and stop when a page comes back with fewer products than you asked for. pagination.totalRecordsFound tells you how many there are in all, so a store can show “page 3 of 7”.

const PAGE_SIZE = 50;
const products = [];
let pageNumber = 1;
let total;

for (;;) {
  const res = await fetch(
    `https://api.dyj.one/v1/market/feeds/${FEED_ID}/items/search`,
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.DYJ_SECRET_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ pageNumber, pageSize: PAGE_SIZE }),
    },
  );
  if (!res.ok) throw new Error(`Feed sync failed: ${res.status}`);

  const page = await res.json();
  if (total !== undefined && page.pagination.totalRecordsFound !== total) {
    // The feed changed under the walk. Start over rather than stitch two feeds together.
    products.length = 0;
    pageNumber = 1;
    total = undefined;
    continue;
  }
  total = page.pagination.totalRecordsFound;
  products.push(...page.data);
  if (page.data.length < PAGE_SIZE) break;
  pageNumber += 1;
}

A feed is a live query, so a product leaving between two of your page reads shifts every later product up one slot, and one at the boundary can be skipped. Walk the pages promptly rather than over hours, and compare totalRecordsFound between pages: if it moved, the feed moved, and the cleanest answer is to start the walk again. Membership is recomputed on every request, so a re-run costs nothing but time.

Cadence

Hourly is plenty for most catalogues; daily is fine for a slow-moving one. Responses carry Cache-Control: private, max-age=60, which is the floor worth respecting — polling faster than once a minute returns the same answer and spends your rate limit.

Products that disappear

A piece leaves your feed when the designer delists it, when their plan lapses, when an admin suspends it, when they block your business, or when a re-price moves it out of your cost band. You are not asked to approve any of that, and none of it is an error.

Handle it by treating each full sync as authoritative: anything you hold that is not in the results is no longer orderable. Deactivate rather than delete — products come back, and keeping your own id stable means a returning piece keeps its reviews, its URL and its SEO history.

You will be told, too

When a product leaves a feed you carry, DYJ raises a notification in your dashboard — one per product, and one per business rather than one per feed if you run several. New products arriving are deliberately silent: that is the ordinary operation of an automatic feed, and announcing each one would bury the notices that matter.

Prices

Save the entire approved product together: specifications, price, production terms, photos and embed fields. updatedAt advances on publication and repricing;specificationVersion changes only when the approved specifications change. A pending creator edit leaves the current publication intact. Refresh even when the specification version is unchanged: price and your feed margins may still change.

price.totalPrice is recomputed on every read from the designer’s current cost (price.listingPrice) and your current margin schedule. Store it with your product, refresh it on each sync, and do not hard-code it into anything long-lived. If a price moves more than you expect, the cost moved: a change there can move a piece into a different rule or a different tier of the same schedule — the feed preview in your dashboard shows which rule priced each piece.