OpenActive Developers
Data ValidatorDataset DashboardW3C Community Group
  • Welcome to our community
  • Publishing Data
    • Data Feeds
      • How an RPDE data feed works
      • Types of RDPE feed
      • Implementing RPDE
      • Testing RPDE feeds
      • Scaling RPDE feeds
    • Activity list references
    • Including geo coordinates
    • Schedules
    • Dataset Sites
    • Virtual Events
    • On-Demand Events
    • Opening Hours
    • Data Quality
  • Using data
    • Harvesting opportunity data
      • Large Integers in JavaScript
    • Tutorial: Consuming an RPDE feed
    • Attribution
  • Open Booking API
    • Key Decisions
    • Implementing booking
    • Testing booking
      • Configuring Test Suite
      • Implementing the Test Interface
        • Test Interface Actions
        • Create Opportunity Endpoint
      • Random Mode: Generating Test Opportunity Data
      • Running Test Suite
      • Generating the Conformance Certificate
  • Data Model
    • Data Model Overview
    • @context and JSON-LD
    • Types Reference
      • Action
      • AudioObject
      • BabyChanging
      • Barcode
      • BookingService
      • BooleanFormFieldSpecification
      • Brand
      • ChangingFacilities
      • ConceptScheme
      • Concept
      • CourseInstance
      • Course
      • Creche
      • CustomerAccount
      • DataCatalog
      • DataDownload
      • Dataset
      • DropdownFormFieldSpecification
      • DynamicPayment
      • Entitlement
      • EventSeries
      • Event
      • FacilityUse
      • FileUploadFormFieldSpecification
      • GeoCoordinates
      • HeadlineEvent
      • ImageObject
      • IndividualFacilityUse
      • InternalApplicationError
      • InternalLibraryConfigurationError
      • InternalLibraryError
      • Lease
      • LocationFeatureSpecification
      • Lockers
      • MediaObject
      • OfferOverride
      • Offer
      • OnDemandEvent
      • OpenBookingError
      • OpeningHoursSpecification
      • OrderItem
      • OrderProposal
      • OrderQuote
      • Order
      • Organization
      • ParagraphFormFieldSpecification
      • Parking
      • PartialSchedule
      • Payment
      • Person
      • Place
      • PostalAddress
      • PriceSpecification
      • PrivacyPolicy
      • PropertyValueSpecification
      • PropertyValue
      • QuantitativeValue
      • Schedule
      • ScheduledSession
      • SessionSeries
      • ShortAnswerFormFieldSpecification
      • Showers
      • Slot
      • SportsActivityLocation
      • TaxChargeSpecification
      • TermsOfUse
      • Terms
      • Toilets
      • Towels
      • VideoObject
      • VirtualLocation
      • WebAPI
  • Specifications
    • Specifications Overview
  • Useful links
    • Data Visualiser
    • Data Validator
    • Dataset Dashboard
    • Non-technical Guidance
  • OpenActive on GitHub
    • Overview
    • Activity List
    • Community
    • Controlled Vocabularies
    • Dataset Publication
    • Documentation
    • Implementation Support
    • Programmes
    • RPDE
    • SKOS
    • Specifications
    • Validators
Powered by GitBook
On this page
  • Recommended Approach
  • Example Code
Edit on GitHub
  1. Using data
  2. Harvesting opportunity data

Large Integers in JavaScript

PreviousHarvesting opportunity dataNextTutorial: Consuming an RPDE feed

Last updated 1 year ago

As mentioned in , the modified field can be expected to have values too large to be able to be precisely represented by JavaScript's type.

In some cases, this may be no issue. If your RPDE feed harvesting implementation does not do anything with the modified field, then that's fine. However, if your implementation involves asynchronous and concurrent processing in such a way that individual RPDE items can be stored not strictly in the order that they appeared in the feed, you will need to use modified comparisons in order to determine which RPDE item is the most up to date.

If doing this, the recommended approach is to:

Recommended Approach

  • Parse the modified from the RPDE page using a custom JSON parser, that can handle large integers — we recommend .

  • Keep it as a string in memory.

    • NOTE: The recommended approach is to keep it in memory as a string and NOT a . This is because most of the JavaScript ecosystem cannot handle BigInts. In a sufficiently complicated app, where you have HTTP requests, database integration, logging, file system read/writes, etc, each one of these integration points will break as soon as it encounters a BigInt. Experience shows that this can be much harder to maintain and so we instead recommend storing this data as strings and temporarily using BigInts for numeric comparison only.

  • If comparing the modified of two different RPDE items, use JavaScript's type for the numeric comparison e.g. BigInt(aModified) >= BigInt(bModified).

  • When storing to a database, store with the database's 64-bit numeric type (bigint in PostgreSQL).

Example Code

Here's some example code demonstrating fetching an RPDE page with :

const res = await axios('https://example.com/rpde/scheduled-sessions', {
  // Ensure that fidelity is not lost in the JSON parsing process.
  transformResponse: (res) => jsonParseConvertingTooLargeInt(res),
});
// Store `modified`s as strings
const rpdeItemsWithStringModifieds = res.data.items.map((item) => ({
  ...item,
  modified: String(item.modified),
});
for (const rpdeItem of rpdeItemsWithStringModifieds) {
  const existingItem = rpdeItemCache[rpdeItem.id];
  // Temporarily use the BigInt type for numeric comparison
  if (existingItem && BigInt(rpdeItem.modified) < BigInt(existingItem.modified)) {
    continue;
  }
  rpdeItemCache[rpdeItem.id] = rpdeItem;
}
 
// --- A DIFFERENT MODULE - JSON PARSING UTILS ---
const { parse, isInteger } = require('lossless-json');

/**
 * @param {string} s
 */
function jsonParseConvertingTooLargeIntsToBigInts(s) {
  return parse(s, null, convertNumberToBigIntIfTooLargeInt);
}

/**
 * This is called for ALL numeric fields in the response — not just `.modified`.
 * So, we only use custom de-serialization if the value is out of normal integer
 * bounds.
 * There is a possibility that fields other than `.modified` will be caught by
 * this function as there are other integer fields in Opportunity data. This is
 * we transform, just for the de-serialization process, these values to BigInt
 * rather than string, which will make it easier to deal with cases in which
 * another numeric field (e.g. remainingUses) is surprisingly a very large
 * integer.
 *
 * @param {string} value
 * @returns {number | bigint}
 */
function convertNumberToBigIntIfTooLargeInt(value) {
  if (isInteger(value)) {
    const asInt = parseInt(value, 10);
    /* Note we consider equality to either of the bounds to be "too large" just
    to be extra cautious against the effects of precision loss */
    if (asInt >= Number.MAX_SAFE_INTEGER || asInt <= Number.MIN_SAFE_INTEGER) {
      return BigInt(value);
    }
    return asInt;
  }
  return parseFloat(value);
}

Things to note about the above example:

It uses , which has been proven to work for this issue (problems with JSON.parse's reviver and with the library are discussed within ).

It uses both strings and BigInts at various points to handle the data. Each of these is explained in comments, but broadly it uses the rules defined in the .

lossless-json
json-bigint
this GitHub issue
recommended approach
RPDE
number
lossless-json
BigInt
BigInt
axios
Storing RPDE modified with less than 64-bit integers