Ch 01 Stopping bad book data Revised

Chapter 01

Stopping bad book data before it reached retailers

A rule engine that checks book metadata against the ONIX 3.0 spec before it reaches retail partners, and a way to fix each problem where it’s reported.

Perfect Bound · The Stable Book Group · 2 min read

The problem

Publishers on Perfect Bound send their book records to retail and distribution partners in a public ONIX feed. They could push records into that feed with missing or contradictory metadata. Errors showed up only as a vague toast message, bad records reached retail partners, and some partners can’t ingest a record at all if it lacks key fields.

What I built

I built a rule engine that classifies each issue as a blocker, which stops the feed from being enabled, or a warning, which is advisory. The rules follow the EDItEUR ONIX 3.0 specification and the BISG guidelines for the US trade. The engine collects every issue in one pass.ChoseEvery issue at once, so publishers see the whole list. RejectedStopping at the first error.

The rules check required fields (an ISBN-13 with a valid checksum, title, product form, status, publication date and publisher), contradictions between fields (juvenile and adult subject codes on the same book, a publication date that doesn’t fit the status, an audience that doesn’t fit the subject), and whether prices and age ranges are complete.

I replaced the toast with an issue panel that stays on screen. Each issue has a Fix action that opens a modal with just the fields at fault, so they can be corrected in place. The modal runs a live duplicate-ISBN check using the same checksum logic as the backend.ChoseThe backend’s checksum logic, so the form and the server agree on what a valid ISBN is.

When the last blocker is fixed, one Save & Enable action finishes the job.ChoseBlockers stop a feed from being enabled, but never from being disabled, so a publisher can always turn a feed off.

Enabling a feed writes to two tables, and those writes run in a single transaction.ChoseOne transaction, so the two tables can’t drift apart.

While building this I found a latent bug. Request validation was stripping a field, so every per-receiver ONIX rule, anything other than the global default, was silently ignored. I fixed it.

To roll the change out, I wrote a backfill that runs as a dry run by default. It reports the affected records for each type of issue before it changes anything.ChoseDry run by default, so the report of affected records comes before any change.

After rollout, 178 of the blocked formats turned out to belong to a publisher using valid product codes that aren’t ISBNs: GTIN-13s outside the ISBN ranges. I added a per-account opt-in, which unblocked them.ChoseThe opt-in is read from account data already loaded, because the validator runs once per format across whole catalogs. RejectedA query per format.

The logic that classifies a product code is shared by the validator, the feed generator and the UI.ChoseOne classification in one place, so the validator, feed generator and UI can’t disagree.

Outcome

At rollout, the backfill’s dry run found 3,507 formats switched on for the feed that failed at least one blocker, across 177 publisher accounts. Publishers now see every issue before a record can be enabled, and fix each one where it’s reported.

Stack

  • TypeScript
  • Node.js
  • Express
  • Sequelize
  • MySQL
  • Next.js
  • React
  • react-hook-form
  • zod
  • Jest