OpenActive Developers
Data ValidatorDataset DashboardW3C Community Group
  • Welcome to our community
  • Publishing Data
    • Data Feeds
      • How an RPDE data feed works
      • Types of RDPE feed
      • Implementing RPDE
      • Testing RPDE feeds
      • Scaling RPDE feeds
    • Activity list references
    • Including geo coordinates
    • Schedules
    • Dataset Sites
    • Virtual Events
    • On-Demand Events
    • Opening Hours
    • Data Quality
  • Using data
    • Harvesting opportunity data
      • Large Integers in JavaScript
    • Tutorial: Consuming an RPDE feed
    • Attribution
  • Open Booking API
    • Key Decisions
    • Implementing booking
    • Testing booking
      • Configuring Test Suite
      • Implementing the Test Interface
        • Test Interface Actions
        • Create Opportunity Endpoint
      • Random Mode: Generating Test Opportunity Data
      • Running Test Suite
      • Generating the Conformance Certificate
  • Data Model
    • Data Model Overview
    • @context and JSON-LD
    • Types Reference
      • Action
      • AudioObject
      • BabyChanging
      • Barcode
      • BookingService
      • BooleanFormFieldSpecification
      • Brand
      • ChangingFacilities
      • ConceptScheme
      • Concept
      • CourseInstance
      • Course
      • Creche
      • CustomerAccount
      • DataCatalog
      • DataDownload
      • Dataset
      • DropdownFormFieldSpecification
      • DynamicPayment
      • Entitlement
      • EventSeries
      • Event
      • FacilityUse
      • FileUploadFormFieldSpecification
      • GeoCoordinates
      • HeadlineEvent
      • ImageObject
      • IndividualFacilityUse
      • InternalApplicationError
      • InternalLibraryConfigurationError
      • InternalLibraryError
      • Lease
      • LocationFeatureSpecification
      • Lockers
      • MediaObject
      • OfferOverride
      • Offer
      • OnDemandEvent
      • OpenBookingError
      • OpeningHoursSpecification
      • OrderItem
      • OrderProposal
      • OrderQuote
      • Order
      • Organization
      • ParagraphFormFieldSpecification
      • Parking
      • PartialSchedule
      • Payment
      • Person
      • Place
      • PostalAddress
      • PriceSpecification
      • PrivacyPolicy
      • PropertyValueSpecification
      • PropertyValue
      • QuantitativeValue
      • Schedule
      • ScheduledSession
      • SessionSeries
      • ShortAnswerFormFieldSpecification
      • Showers
      • Slot
      • SportsActivityLocation
      • TaxChargeSpecification
      • TermsOfUse
      • Terms
      • Toilets
      • Towels
      • VideoObject
      • VirtualLocation
      • WebAPI
  • Specifications
    • Specifications Overview
  • Useful links
    • Data Visualiser
    • Data Validator
    • Dataset Dashboard
    • Non-technical Guidance
  • OpenActive on GitHub
    • Overview
    • Activity List
    • Community
    • Controlled Vocabularies
    • Dataset Publication
    • Documentation
    • Implementation Support
    • Programmes
    • RPDE
    • SKOS
    • Specifications
    • Validators
Powered by GitBook
On this page
  • Libraries
  • Dataset discovery
  • Harvesting feeds
  • Considerations
  • Common Pitfalls
Edit on GitHub
  1. Using data

Harvesting opportunity data

PreviousData QualityNextLarge Integers in JavaScript

Last updated 6 months ago

Libraries

Although the logic to harvest and use OpenActive data is straightforward, there are several libraries that provide helpers to aid the consumption of opportunity data feeds.

The table below lists these libraries:

Language
Dataset Discovery
Harvesting Feeds

JavaScript / TypeScript

N/A

Python

Ruby

* Note that the Ruby library before it can be used for dataset discovery

Dataset discovery

As described in the data catalogue , OpenActive datasets can be discovered automatically by "spidering" links within the canonical file.

Harvesting feeds

Considerations

  • Combining feed pairs

    • Ensure that updates and deletes from both parent and child feeds are considered (such as SessionSeries/ScheduledSession or FacilityUse/Slot - see for more information)

  • Harvest frequency

    • To ensure your resources are not wasted, especially as you scale feed consumption, ensure that modes are respected (i.e. wait 8 seconds if there are no items in the feed before making the next request). Due to caching, more frequent requests will simply hit a CDN and return the same response, so there is no advantage in polling faster than this.

  • De-serializing RPDE Feeds:

    • Ensure that the data type used to de-serialize the modified timestamp can support signed 64-bit integers. .

  • Resyncs

    • Consuming an RPDE feed from the beginning is termed a "resync".

    • RPDE feeds are not designed to be resynced frequently.

    • The feed consumer must continue to consume updates from the end of the feed to ensure the data stays up-to-date, rather than downloading all data from each update.

    • Resyncing any individual feed more than once each week is not recommended, as it increases the load on the open data publisher's servers, which will likely result in a high number of 429 responses and could cause your IP address to be blacklisted.

Common Pitfalls

Storing RPDE `modified` with less than 64-bit integers

This data type has been seen to generate integers up to values of 2⁶⁰.

Therefore, it is recommend to implementers to use data types that can store at least a signed 64-bit integer with precision.

Language specific guidelines:

  • JavaScript / TypeScript:

  • C#: Use a long as opposed to an int

  • Other languages: Ensure that the integer type that is being used to de-serialize the modified timestamps from RPDE feed pages has at least as much size and precision as a signed 64-bit integer.

*

A common approach to creating modified values for an RPDE feed is to use SQL Server's timestamp/rowversion data types. This approach is suggested in the .

JavaScript are 64-bit floating point numbers, which means that integers cannot be represented with sufficient precision beyond 2⁵³.

Getting around this is a bit complicated, but possible. See this page for more info: .

RPDE specification
numbers
Large Integers in JavaScript
@openactive/dataset-utils
openactive-python
openactive-python
openactive.rb
openactive.rb
requires updating
processing guidance
OpenActive Data Catalog Collection JSON-LD
Types of RPDE feed
More info here
sleep and live