In March 2020, BookNet Canada’s Bibliographic Committee reviewed and endorsed a countdown calendar to help guide Canadian data providers as they transition to ONIX 3.0. The impetus for this was Amazon’s mandate for ONIX 3.0 feeds for print books by the end of calendar year 2020. The committee endorsed August 28, 2020 as the date by which Canadian data providers should have been able to generate ONIX 3.0 records and feeds, as well as sending those to data recipients who chose to accept them. We updated you on the status of the Canadian market’s transition to 3.0 in our September blog post, and now we’re back to give you more!
It’s December 22. Do you know where your ONIX 3.0 records are?
In our September blog post, we shared numbers from BiblioShare, BookNet Canada’s quality-controlled aggregation system that collects and disseminates bibliographic data using ONIX — in both ONIX 2.1 and ONIX 3.0 formats. At that time, the size of the BiblioShare 3.0 dataset was approximately 10% of the size of the 2.1 dataset, amounting to approximately 300,000 total 3.0 records. We attributed this low figure to two factors:
most of the large data suppliers in Canada — namely distributors handling multiple publishers' data as well as the large multinational firms — were working on timelines to match Amazon’s year-end deadline (now extended to end of March 2021), not the Canadian Bibliographic Committee’s deadline of Aug. 28, 2020; and
most data suppliers were not going to invest resources into mapping non-active (ex. out of print) titles to 3.0, so the volume of titles we were expecting to see in 3.0 were limited to active and forthcoming releases.
The difference a quarter makes
At the end of November, the number of ONIX 3.0 files in our BiblioShare database makes up almost 12% of the 3.5 million 2.1 records, amounting to approximately 420,000 records. This isn’t a notable change from our count back at the end of August — why, you ask?
A number of publishers and distributors have identified some issues with their feeds and are working out the kinks. Some of those kinks are documented below, and many can be categorized as schema validation issues.
Schema validation
Schema validation is an important step that's often overlooked. Quality-controlled ONIX databases require that all files pass schema validation before they’re loaded into the database to maintain a base level of quality in the dataset.
Aggregators often have to fix mistakes in ONIX files before they’re loaded into their database, with minor mistakes fixed on the fly based on a reasonable guess that may not reflect publisher preferences. We believe that control should remain in the hands of the data sender, and part of having that control is validating your files so that you can find any existing problems and fix them your way. In short, doing your own XML validation will help you stay in control of your data. Plus, an invalid file won't load into the database and will have to be fixed.
You wouldn't publish a book without proofreading it first and Schema validation is the same principle — proofreading for computer loadability. Own your data! Schema validation is quick and easy, and if you need help getting started or learning more about the issues that might be leading to file invalidation, we have some excellent resources to get you caught up:
In our blog
In our user documentation
Some pain points
In the three months since our last update, suppliers making the transition continued to report difficulties mastering two key components that differentiate 3.0 from 2.1: collections and markets.
To support your collections education, BNC’s Bibliographic Manager Tom Richardson crafted three guides to sort it all out:
to start, learn how to group products using the collections composite;
next, tackle the title detail composite; and
finally, tie it all together.
If you need to brush up on your comprehension of markets, refer to this essential webinar on territories and markets.
But it’s not all sunny in ONIX 3.0 land. Tom Richardson has been tirelessly documenting common errors he’s observing in 3.0 feeds sent to BiblioShare. Here are some of the common issues and the problems they pose:
| Issue | Downstream impact |
|---|---|
| Codes only allowed in one version of ONIX appear in the other | Forces modification of data by data recipient, and depending on code and time, these are either replaced with a near equivalent or proprietorial value (preferred to losing data) OR deleted. |
| Selective use of ONIX and its Code Lists: either partial lists (inexplicably not allowing use of codes) or not supporting a composite in its entirety | The devil or diversity is in the details. You can’t say what you mean when you don’t use the best available code — or when you're trying to describe a book without the help of the section that covers it best. Granularity doesn’t grind you down it builds you up to better meanings. |
| Poor expression or no support for:
◾Market in Product Supply / Use of Sales Rights as proxy statement for distributors ◾ Market level status & dates |
Given the increasingly complex international marketing and distribution of physical books (consider: multiple international distributors, different publication dates in different markets, etc.) support for multiple markets in <SupplyDetail> (where each market is a repeat of <ProductSupply>) is becoming increasingly important. And for ebooks, differing arrangements with different retail platforms should be handled in the same way. |
| Notification Type "04" update use triggering block update processing | Notification Type 04 is reserved for block updates. When used on a regular non-block file it can trigger unnecessary processing routines. |
| Duplication of Record Reference and Product Identifier ("Block zero") provided for two records | Surprisingly common and found in data from a major aggregator for multiple data senders, this fails to differentiate the two records (i.e. products) from each other. |
| Misunderstanding how to support multilingual metadata in ONIX, seen in repeating whole composites instead of repeating an individual element within composites | |
| Incoherent or nonsensical data pairings (ex. Product Form “BB” paired with Product Form Description “Print PDF,” “battery weight” associated with normal book codes) | Likely a result from bad mapping (ICYMI: read Of Metadata and mapping and why you cannot convert ONIX 2.1 to 3.0), this has the potential to confuse and/or mislead both data recipients as well as consumers. |
All this to say…
We’re here to help
With your ONIX 3.0 feed in hand — erm, in the interwebs — your next step, if you haven’t done so already, is to establish a 3.0 field with BiblioShare. As easy as sending a 2.1 feed, sending your ONIX 3.0 feed to BiblioShare affords you the opportunity to have your file reviewed by BNC’s experts and validated by the best in the business!
Not sure how to send ONIX 3.0 files to BiblioShare? Problem solved.
If you’re still in the midst of transitioning to ONIX 3.0, you’re not alone. Transitioning is no small undertaking. If you’re struggling, have questions, or just need a friendly ear, don’t hesitate to email us. BookNet is Canadian publishing’s standards organization, your standards organization. We’re here to help. How can we help you best?
Reminder: don’t let the sun go down on 2.1
Transitioning to 3.0 doesn’t mean your company will immediately sunset your 2.1 feed. We at BookNet expect data suppliers will need to support dual ONIX 3.0 and 2.1 feeds through 2021. Just because some of your trading partners are asking for and can support 3.0 doesn’t mean you can ignore those partners who haven't made the transition.
Who joined the winner’s circle?
The following publishers transitioned to 3.0 and established feeds to BiblioShare in the months since September 1: Cengage, New Society, Optimum Publishing, Penn State University, and University of South Carolina.
Crossing the finish line by the August 28 deadline: Annick Press, Anvil Press, Arbeiter Ring Publishing, Athabasca University Press AU Press, Banff Centre Press, Between the Lines, Book*hug Press, Books We Love, Boulder Books, Breakwater Books, Ltd, Brick Books, Broadview Press, Brush Enterprises Inc, Caitlin Press, CAMH The Centre for Addiction and Mental Health - Publication Services, Cape Breton University Press, Child's Play, Chouette Publishing, Coach House Press, Compass Maps (PopOut Maps), Cook By Colours Inc, Cormorant Books, Coteau Books, Crow Cottage Publishing, Demeter Press, Dundurn, Elsevier Science & Technology Books, Engage Books, Flanker Press, Freehand Books, Goose Lane Editions, House of Anansi, Iguana Books, Inanna Publications, Infinite Ideas, Ingram Content Group, Inhabit Media Inc, Invisible Publishing, Irwin Law Inc., ISER Books, Joseph Norman Editions PD Meany Publishers, Legend Press, Linda Leith Publishing, Mansfield Press, McGill-Queens University Press, Memoirs Publishing, NeWest Press, Oolichan Books, Owl Kids, Pajama Press, Panoma Press, Pembroke Publishing Ltd., Portage & Main Press, Ronsdale Press, Scholastic Ebooks, Second Story Press, Signature Editions, Star Crossed Press, Talonbooks, The MIT Press, Thistledown Press, Thomas Allen, Three O'Clock Press, Turnstone Press, Tyche Books, UBC Press, United Church Publishing House UCPH, University of Alberta Press, University of Calgary Press, University of Manitoba Press, University of Ottawa Press, Vehicule Press, Whitecap Books Ltd., Wordcatcher Publishing Group, and Wordsworth Editions.
Virtual high fives!
Insights into the behaviour of Canadian book borrowers.