# Welcome to our community ## Help - I'm not a developer! Here's a quick [escape hatch](https://www.openactive.io/getting-started/) - quick, run! But before you do, e-mail this page to your development team. ## What is OpenActive? If you've just been sent this link without any context, check out this video before you go any further: {% embed url="" %} You can find out more at OpenActive.io {% endembed %} ## Are you on Slack yet? The OpenActive community are all available on Slack, [click here](https://openactive.io/public-chat/) to find out more. We'll be happy to help with any issues you have during your implementation, or hear any feedback. # Data Feeds ## Overview OpenActive is an initiative that promotes the use of open data to help people become more active. It aims to make opportunities to be active as discoverable as possible, similar to how one might find a hotel room or flight online. OpenActive is primarily concerned with feeds of "opportunity data" – that is, information about what, where and when activities are taking place. To take advantage of the OpenActive ecosystem of data users, data must be published via open data feeds. OpenActive specifications can be used to publish three types of physical activity opportunity: * Sessions and facility listings, which may also be bookable * Organiser listings or directories * Route guides ## Listing-only vs Bookable Opportunity Data Feeds For opportunities with scheduled dates and times, OpenActive supports two levels of user interaction, and a data publisher must choose which level they wish to implement: 1. **Listing-only Opportunity Data**: * Data that includes near-real-time up-to-date details about a physical activity opportunity that can be discovered by potential participants. Listing data typically includes information like the type of activity, time, location, price, provider and availability. * The main goal of this data is to allow potential participants to find opportunities that match their interests and availability, and are actually happening. * However, the listing data itself doesn't enable users to reserve or book a place in the activity directly. It's more of an informative listing that helps people discover available opportunities. * **Implementation requirements:** opportunity data RPDE feeds that include deep-link URLs to the opportunity within the listing or booking system. 2. **Bookable Opportunity Data**: * This refers to data and functionality that allows potential participants to not only discover an activity but also to reserve or book a place directly through a platform or application. * It enables integration with booking systems to allow real-time availability checking and booking. * Offering bookable data can lead to an increase in participation since it makes the process more streamlined for users — they can find and book an activity all in one place. * **Implementation requirements:** opportunity data RPDE feeds and the Open Booking API. In summary, while "listing-only" data offers up-to-date information about physical activity opportunities, "bookable" data takes it a step further by allowing direct reservations or bookings. The implementation of both provides a seamless experience from discovery to participation. These feeds must be published using the [OpenActive Real-time Paged Data Exchange (RPDE) specification](https://openactive.io/realtime-paged-data-exchange/), which provides a mechanism to allow systems to synchronise database state in near-realtime. ## Organiser Data Feeds Activity organisers (e.g. local sports clubs) are crucial entry points for participation. Many do not yet manage their sessions or facility slots online, or are not in a position to publish individual scheduled opportunities. We are piloting new data feeds to boost their discoverability and attract new participants. National Governing Bodies, maintaining lists of accredited clubs, are a potential source of quality open data feeds. This data typically includes information about the organisation itself, primary activities, location, contact details, operating hours, membership, and facilities. ## Route Guide Data Feeds We are piloting new data feeds to support activities such as walking, running, cycling, or paddling. Such data can be consumed by platforms that support navigation, social sharing, and peer-to-peer route discovery. Route guide data can include geographical information, difficulty, duration estimates, accessibility details, points of interest, and user-generated feedback/reviews. ## Primary sources Opportunity data must always be published from **primary sources** (e.g. the booking system that an organiser uses to manage their own activities) rather than **secondary sources** (e.g. a third-party survey of organisers or web form which captures the sessions they are currently running) Publishing data from primary sources rather than secondary sources is essential for several reasons: 1. **Accuracy**: Primary sources provide data directly from the organiser or provider of opportunity, which means it's more likely to be accurate and up-to-date. Secondary sources such as surveys and forms are usually not thoroughly reviewed by providers. Much less care and attention is taken when filling in these forms than on the activity provider's own website and booking system. 2. **Timeliness**: Activities, classes, and schedules often change. Primary sources like booking systems are updated in real-time, so the information is always current. Relying on secondary sources would mean periodic data collection, which can become quickly outdated. 3. **Scalability**: As the number of activity providers grows, it becomes increasingly expensive for secondary sources to maintain a large set of data. Primary sources do not require any such central maintenance overhead. 4. **Data richness**: Primary sources can often provide a richer dataset, including details about availability, pricing, venue facilities, instructor qualifications, and more. This can offer a more complete picture for end-users, helping them make informed decisions. 5. **Reduced burden**: For activity providers, integrating their primary booking or management systems once with OpenActive can reduce the burden of repeatedly reporting data or filling out surveys. It's a "set it and forget it" approach, which is more efficient in the long run. 6. **Trustworthiness**: Data users and consumers will trust data more when they know it's coming directly from the source and is current. This trustworthiness is likely to increase the adoption of applications and platforms that use OpenActive data, and increase consumer confidence of data from our sector. In conclusion, to provide real-time, accurate, and comprehensive data about scheduled physical activity opportunities, relying on primary sources is much more effective and efficient. That said, the OpenActive ecosystem is evolving and we recognise there are other valuable data types - like clubs and routes - that can help potential participants find and plan physical activity. {% content-ref url="/pages/-LONU-bHf0VDp9dDdbxv" %} [How an RPDE data feed works](/publishing-data/data-feeds/how-a-data-feed-works) {% endcontent-ref %} # How an RPDE data feed works ## RPDE feeds explained The following video is a beta explainer animation for Realtime Paged Data Exchange (RPDE) feeds. If you find this short video useful, please drop any feedback to , and we will extend it into a full explanation. For a full explanation of how to create an RPDE feed, please see the [specification](https://www.openactive.io/realtime-paged-data-exchange/#paging). {% embed url="" %} ## Must Read: Principles of Feeds ### Items must never disappear It's important to ensure that items don't disappear from the feed without first passing through the `deleted` state, as if this happens they will live on forever in the downstream apps and services, which will have cached the previously published version indefinitely. See the [deleted items](https://www.openactive.io/realtime-paged-data-exchange/#deleted-items) section of the specification for more information. ### The query must be followed exactly Misreading the query in the specification is the single most common cause of incorrect implementation. Please read it carefully and ensure that brackets and comparators are used correctly. `>` not `>=` for example. Please ensure that you have implemented [this query](https://www.openactive.io/realtime-paged-data-exchange/#sql-query-example-for-timestamp-id) correctly: ```sql --include WHERE clause only if @afterTimestamp and @afterId provided WHERE (modified = @afterTimestamp AND id > @afterId) OR (modified > @afterTimestamp) ORDER BY modified, id ``` # Types of RDPE feed ## Time-based Events: **Regular Classes and Sessions** These always have at least `activity`, `location` and `startDate` specified: so a Yoga class in Downtown Leisure Centre, at 7pm, on a Tuesday. See [here](/data-model/data-model-overview#classes-and-sessions) for further clarification of the types available. The OpenActive Modelling Specification 2.0 represents regular events using a hierarchy of types: [`EventSeries`](https://www.openactive.io/modelling-opportunity-data/#grouping-together-events-eventseries-), [`SessionSeries`](https://www.openactive.io/modelling-opportunity-data/#regular-sessions-sessionseries-and-scheduledsession-), and [`ScheduledSession`](https://www.openactive.io/modelling-opportunity-data/#regular-sessions-sessionseries-and-scheduledsession-), linked via the `superEvent` and `subEvent` properties. These are described by example in the diagram below: ![](https://docs.google.com/drawings/u/0/d/s78NrrrLPOgQHs-TaXftQdg/image?w=602\&h=268\&rev=325\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) ### **Property inheritance** The `ScheduledSession` will inherit the properties of the `SessionSeries`, and the `SessionSeries` will inherit the properties of the `EventSeries`. This means that if an `EventSeries` is not supplied, its details must be included on the `SessionSeries`. ![](https://docs.google.com/drawings/u/0/d/sC3OM5Gc26L84_nUVy7uWhA/image?w=602\&h=181\&rev=2\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) ### **Property inheritance overrides** The OpenActive Modelling Specification 2.0 represents regular events using a hierarchy of types: `EventSeries`, `SessionSeries`, and `ScheduledSession`. These are described by example in the diagram below: ![](https://docs.google.com/drawings/u/0/d/sNuN0HrwIIJduzHZ-_PIMCw/image?w=602\&h=268\&rev=24\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) ### **Permissible configurations** A `SessionSeries` must always be supplied, including details that would otherwise be present in an `EventSeries` if no `EventSeries` is supplied. In order for data to be [**bookable**](https://www.openactive.io/open-booking-api/EditorsDraft/#dfn-bookability) a `ScheduledSession` must be supplied, either embedded, within a separate feed, or generated through a `Schedule`. For large providers with many events that are described identically but occur in the same location or in different locations, an `EventSeries` should be used to group these together. This avoids seemingly duplicate search results from a single provider. ![](https://docs.google.com/drawings/u/0/d/sruVQes8cWV4JuLh6rHxt-A/image?w=602\&h=321\&rev=144\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) ### **Exposing the model in feeds** For systems targeting small providers, the `ScheduledSession` may be embedded in the `SessionSeries` or visa versa. For systems targeting large providers or small providers with high volumes of sessions, the `ScheduledSession` is highly recommended to be provided in a separate feed to the `SessionSeries`. This will reduce data transfer volumes significantly. The `EventSeries` is unlikely to change frequently enough compared with the `SessionSeries` to warrant its own feed, and so can usually be embedded in the `SessionSeries`. For [**bookable**](https://www.openactive.io/open-booking-api/EditorsDraft/#dfn-bookability) data, the possible feed combinations are described in the diagram below: ![](https://docs.google.com/drawings/u/0/d/sVvrks8vK_gfSRejrIIIMGw/image?w=636\&h=206\&rev=363\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) {% hint style="warning" %} Please note that the first two "combined feed" options given above are no longer recommended for new OpenActive implementations. These options increase complexity for data users, and create unnecessary additional load on all systems. {% endhint %} Bookable data feed examples: * Small provider: [SessionSeries with ScheduledSession](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries_example_1.json\&version=2.0) * Small provider (inverted): [ScheduledSession with SessionSeries](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fscheduledsession_example_1.json\&version=2.0) * Small provider (high volume): * [SessionSeries](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries-split_example_1.json\&version=2.0) and [ScheduledSession](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fscheduledsession-split_example_1.json\&version=2.0) (physical) * [SessionSeries](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries-split-virtual_example_1.json\&version=2.x) and [ScheduledSession](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fscheduledsession-split-virtual_example_1.json\&version=2.x) (virtual) * Large provider: [SessionSeries with EventSeries](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries-eventseries-split_example_1.json\&version=2.0), and [ScheduledSession](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fscheduledsession-split_example_1.json\&version=2.0) Listings data feed examples: * Small provider: [SessionSeries](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries-split_example_1.json\&version=2.0) * Large provider: [SessionSeries with EventSeries](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries-eventseries-split_example_1.json\&version=2.0) ### Use of `@id` and `superEvent` for split feeds When referencing data across feeds (such as between [SessionSeries](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries-split_example_1.json\&version=2.0) and [ScheduledSession](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fscheduledsession-split_example_1.json\&version=2.0)), the value of the **`@id`** must be used. An **`@id`** is a globally unique identifier which must be in URL format for the purposes of namespacing. The **`@id`** does not need to resolve to a functional endpoint, but must use a domain name controlled by the organization or system publishing the data. See [here](/data-model/context-and-json-ld#contexts-properties-and-types) for more information. So for a minimal implementation simply invent a URL pattern that includes your domain for use as your **`@id`**, such as: > "`https://id.ourparks.org.uk/api/session-series/1234`" For example within a `SessionSeries` feed, the `SessionSeries` **`@id`** is defined for each data item: ```javascript "data": { "@context": "https://openactive.io/", "@type": "SessionSeries", "@id": "https://id.ourparks.org.uk/api/session-series/1234", ... } ``` And within a corresponding `ScheduledSession` feed, that `SessionSeries` **`@id`** is referenced by the `superEvent` property: ```javascript "data": { "@context": "https://openactive.io/", "@type": "ScheduledSession", "@id": "https://id.ourparks.org.uk/api/session-series/1234#/subEvent/C5EE1E55-2DE6-44F7-A865-42F268A82C63", "superEvent": "https://id.ourparks.org.uk/api/session-series/1234", ... } ``` ## Time-based Events: Ad-hoc Events The OpenActive model allows for ad-hoc events to be described using the pattern below. Ad-hoc events must only be used to describe truly ad-hoc events, and not to describe regular events such as those described in the previous section. See [here](/data-model/data-model-overview#events) for further clarification of the types available. ### **Summary of ad-hoc event types** The OpenActive Modelling Specification 2.0 represents ad-hoc events using a hierarchy of types: `EventSeries` and `Event`, linked via the `superEvent` and `subEvent` properties. These are described by example in the diagram below: ![](https://docs.google.com/drawings/u/0/d/soe-8CJhGcRo3o-w3Njk_4g/image?w=602\&h=168\&rev=64\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) ### **Property inheritance** The Event will inherit the properties of the `EventSeries`. This means that if an `EventSeries` is not supplied, its details must be included on the `Event`. ![](https://docs.google.com/drawings/u/0/d/sQajrOUrhJ4eXWsfayekEtA/image?w=602\&h=94\&rev=14\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) ### **Permissible configurations** An Event must always be supplied, including details that would otherwise be present in an `EventSeries` if no `EventSeries` is supplied. For large providers with many events that are described identically but occur in the same location or in different locations, an `EventSeries` should be used to group these together. This avoids seemingly duplicate search results from a single provider. ![](https://docs.google.com/drawings/u/0/d/skh7gEfjLUh9HTcLbH2RItw/image?w=259\&h=205\&rev=14\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) ### **Exposing the model in feeds** Systems must include `Event`s in stand-alone feeds separate from `ScheduledSessions` and `SessionSeries`. The `EventSeries` is unlikely to change frequently enough compared with the `Event` to warrant its own feed, and so can usually be embedded in the `Event`. For [**bookable**](https://www.openactive.io/open-booking-api/EditorsDraft/#dfn-bookability) data, example feed combinations are described in the diagram below: ![](https://docs.google.com/drawings/u/0/d/sjJOI7mL_qflDQspoKgNUmw/image?w=636\&h=440\&rev=174\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) Bookable data feed examples: * Events only: [Event](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fevent_example_1.json\&version=2.x) ## Slot-based Events: FacilityUses These always have at least `activity` and`location` specified, where the activity can be booked in slots: so a Tennis at Downtown Leisure Centre with slots available hourly from 8am until 8pm. Two different levels of granularity are available: A `FacilityUse` represents "Badminton at Downtown Leisure Centre", where as `IndividualFacilityUse` is "Court 2 in Sports Hall 3 for Badminton at Towntown Leisure Centre". The OpenActive Modelling Specification 2.0 represents slot-based events using a hierarchy of types: `FacilityUse`/`IndividualFacilityUse` and `Slot`, linked via the `facilityUse` and `event` properties. For facilities a publisher must implement the following two independent feeds: * `http://www.example.org/feeds/facility-uses` * [FacilityUse example](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Ffacilityuse_example_1.json\&version=2.x) (including `IndividualFacilityUse`) * `http://www.example.org/feeds/individual-facility-use-slots` * [Slot example](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fslot_example_1.json\&version=2.x) Note that the above examples publish `Slot` availability at the `IndividualFacilityUse` level, which is now recommended for all new feeds. ### Use of `@id` and `facilityUse` for split feeds When referencing data across feeds (such as between [FacilityUse](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Ffacilityuse_example_1.json\&version=2.x) and [Slot](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fslot_example_1.json\&version=2.x)), the value of the **`@id`** must be used. An **`@id`** is a globally unique identifier which must be in URL format for the purposes of namespacing. The **`@id`** does not need to resolve to a functional endpoint, but must use a domain name controlled by the organization or system publishing the data. See [here](/data-model/context-and-json-ld#contexts-properties-and-types) for more information. So for a minimal implementation simply invent a URL pattern that includes your domain for use as your **`@id`**, such as: > "`https://id.bookingsystem.com/api/facility-uses/1402CBP20150217/individual-facility-uses/1`" For example within a `FacilityUse` feed, the `FacilityUse` **`@id`** is defined for each data item, and `IndividualFacilityUse` **`@id`** values are defined within these: ```javascript "data": { "@context": "https://openactive.io/", "@type": "FacilityUse", "@id": "https://id.bookingsystem.com/api/facility-uses/1402CBP20150217", "individualFacilityUse": [ { "@type": "IndividualFacilityUse", "@id": "https://id.bookingsystem.com/api/facility-uses/1402CBP20150217/individual-facility-uses/1", "name": "Main Tennis Court 1" } ], ... } ``` And within a corresponding `Slot` feed, that `IndividualFacilityUse` **`@id`** is referenced by the `facilityUse` property: ```javascript "data": { "@context": "https://openactive.io/", "@type": "Slot", "@id": "https://id.bookingsystem.com/api/facility-uses/1402CBP20150217/individual-facility-uses/1#/event/2018-03-01T10:00:00Z", "facilityUse": "https://id.bookingsystem.com/api/facility-uses/1402CBP20150217/individual-facility-uses/1", ... } ``` ## Other types of Time-based Events: Headline Events and Courses A feed of [`HeadlineEvent`](https://www.openactive.io/modelling-opportunity-data/#headline-events-headlineevent-) can be used to represent whole day or multi-day events, such as mass participation events, family fun days, etc. * The OpenActive Modelling Specification 2.0 represents these events using a hierarchy of types: `HeadlineEvent` (for the overall event) and `Event` (for small events within the overall event) linked via the `superEvent` and `subEvent` properties. * See [here](/data-model/data-model-overview#headline-events) for further clarification, and [here](http://data.britishtriathlon.org/) for an example. A feed of [`CourseInstance`](https://www.openactive.io/modelling-opportunity-data/#courses-courseinstance-) can be used to represent a fixed-length course. * The OpenActive Modelling Specification 2.0 represents these events using a hierarchy of types: `CourseInstance` (for the whole course) and `Event` (for the individual occurrences) linked via the `superEvent` and `subEvent` properties. * See [here](/data-model/data-model-overview#courses) for further clarification, and [here](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fcourseinstance_event_example_1.json\&version=2.x) for an example. ## **Event relationship overview** The following diagram illustrates the relationships between the event types available within the OpenActive Modelling Specification 2.0: ![](/files/-M5N3BbVBQ0EeCenbmwM) Note the use of aliases (e.g. "IndividualFacilityUseSlot") which are useful when referring to a specific type that is being used in a particular context. The relationship between all types is represented via the `superEvent` and `subEvent` properties, with the exception of: * `FacilityUse`/`IndividualFacilityUse` and `Slot`, which are linked via the `facilityUse` and `event` properties * `FacilityUse` and `IndividualFacilityUse`, which are linked via the `aggregateFacilityUse` and `individualFacilityUse` properties. ## **Schema.org type inheritance overview** The model itself (the properties within the types) follows a *different inheritance structure* to the property inheritance structure described above. `EventSeries`, `SessionSeries`, `ScheduledSession`, `HeadlineEvent`, `CourseInstance`, and `Slot` all **sub-class** `Event`. This can be useful for modelling the entities within certain frameworks**.** ![](https://docs.google.com/drawings/u/0/d/sHnIqB65tCLtqxkTO0dBaVA/image?w=602\&h=272\&rev=181\&ac=1\&parent=1C_eO6JC8tt7-K-XiilHzPKXKenjjHiiOS7nCW07tlLk) # Implementing RPDE Always use a JSON library to generate the output, and never construct the JSON manually. Language-specific examples of library implementations that also support the removal of nulls, empty strings and empty arrays - as the specification stipulates - are included here. ## .NET, PHP, Ruby and JavaScript/TypeScript Libraries Several libraries are available that make it really easy to create open opportunity data feeds. The table below lists the available OpenActive libraries:
LanguageOpen Opportunity Data FeedsDataset Site
LanguageOpen Opportunity Data FeedsDataset Site
.NETOpenActive.NETOpenActive.DatasetSite.NET
PHPopenactive/modelsopenactive/dataset-site
Rubyopenactiveopenactive-dataset_site
JavaScript / TypeScript@openactive/models-ts@openactive/dataset-site-template
## Transactions: Preventing delayed item interleaving When concurrent transactions are used to write to tables that power RPDE feeds, the timestamp and change number update must be done outside of the transaction, with high accuracy timestamp, and a "visible" timestamp column added. ### Example of race condition Delayed item interleaving is a race condition that occurs when two concurrent transactions containing "timestamps" or "change numbers" are committed out of order. For example using the [Modified Timestamp and ID](https://www.w3.org/2017/08/realtime-paged-data-exchange/#modified-timestamp-and-id) ordering strategy: * **Transaction 1**: Starts at 10:00:01 and updates RPDE item timestamps * **Transaction 2**: Starts at 10:00:02 and updates RPDE item timestamps * **Transaction 2**: Commits * Data consumer reads the feed at 10:00:03, and gets the latest items up to 10:00:02 * **Transaction 1**: Commits * Items from **Transaction 1** appear in the feed with timestamp 10:00:01, but the data consumer has already moved past them The same issue can be demonstrated with the [Incrementing Unique Change Number](https://www.w3.org/2017/08/realtime-paged-data-exchange/#incrementing-unique-change-number) ordering strategy: * **Transaction 1**: Creates change number 001 and updates RPDE item timestamp * **Transaction 2**: Creates change number 002 and updates RPDE item timestamp * **Transaction 2**: Commits * Data consumer reads the feed up to 002 * **Transaction 1**: Commits * Item from **Transaction 1** appears in the feed with change number 001, but the data consumer has already moved past it ### Preventing the race condition In order to prevent this race condition, one solution is to simply separate the more intensive work of the transaction from the atomic timestamp and change number update, filter out recent changes in the feed, and ensure you are using accurate timestamps: 1. First commit the transaction, then update the timestamps or change numbers after the transaction has been committed as an atomic operation, outside of a transaction, using `GETDATE()` or similar 2. Ensure the RPDE endpoint filters out all items with a "modified" date after 2 seconds in the past, to delay items appearing in the feed 3. If using the [Modified Timestamp and ID](https://www.w3.org/2017/08/realtime-paged-data-exchange/#modified-timestamp-and-id) ordering strategy, use a timestamp column with a high degree of accuracy (e.g. `datetime2` in SQL Server). Updating items in the feed without updating their timestamp/change number immediately does not have any negative effects, as data consumers who are reading the feed will read the updated item earlier than they would otherwise instead of an older version, then read it again after the timestamp/change number is updated as they would normally. Using the RPDE endpoint filter to delay items appearing is a belt-and-braces measure that ensures that [small variances](https://stackoverflow.com/questions/30301302/identity-and-getdate-out-of-order) between timestamp and change number update order are accounted for under high database load, by only presenting data to the data consumer after a small delay, when all timestamp/change number updates have commited. ## C# and .NET Framework ### OpenActive.NET Library We highly recommend using the [OpenActive.NET](https://www.nuget.org/packages/OpenActive.NET/) library for .NET implementations. It includes strongly typed classes for the [OpenActive data model](https://www.openactive.io/modelling-opportunity-data/) including [beta properties](https://www.openactive.io/ns-beta/), and fully compliant serialisation methods too. ### Manual Implementation Use [JSON.NET / Newtonsoft.Json](https://www.nuget.org/packages/Newtonsoft.Json/) with a `ContractResolver` to remove empty strings, and `NullValueHandling` to remove nulls. Empty list values must be removed by setting them to null in the generating code, aided by the extension `ToListOrNullIfEmpty` which is useful at the end of LINQ expressions. ```csharp public class RPDERestUtils { public static HttpResponseMessage CreateJSONResponse(RpdePage page, HttpRequestMessage req) { var e = JsonConvert.SerializeObject(page, Newtonsoft.Json.Formatting.None, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore, ContractResolver = NoEmptyStringsContractResolver.Instance }); var resp = req.CreateResponse(HttpStatusCode.OK); resp.Headers.CacheControl = new CacheControlHeaderValue() { Public = true, // Recommended cache settings from: // https://developer.openactive.io/publishing-data/data-feeds/scaling-feeds MaxAge = page?.items?.Count > 0 ? TimeSpan.FromHours(1) : TimeSpan.FromSeconds(8) }; resp.Content = new StringContent(e, Encoding.UTF8, "application/json"); return resp; } public class NoEmptyStringsContractResolver : DefaultContractResolver { public static readonly NoEmptyStringsContractResolver Instance = new NoEmptyStringsContractResolver(); protected override JsonProperty CreateProperty(MemberInfo member, MemberSerialization memberSerialization) { JsonProperty property = base.CreateProperty(member, memberSerialization); if (property.PropertyType == typeof(string)) { // Do not include empty strings in JSON output (as per OpenActive Modelling Specification) property.ShouldSerialize = instance => { return !string.IsNullOrWhiteSpace(instance.GetType().GetRuntimeProperty(member.Name).GetValue(instance, null) as string); }; } return property; } } public static List ToListOrNullIfEmpty(this IEnumerable source) { if (source != null && source.Count() > 0) return source.ToList(); else return null; } } ``` ## PHP ### OpenActive PHP Models Library We highly recommend using the [OpenActive PHP Models Library](https://packagist.org/packages/openactive/models) for PHP implementations. It includes strongly typed classes for the [OpenActive data model](https://www.openactive.io/modelling-opportunity-data/) including [beta properties](https://www.openactive.io/ns-beta/), and fully compliant serialisation methods too. ### Manual Implementation The specification requires that no null or empty strings are present in the OpenActive feed. To achieve this, define the RPDE response structure as nested arrays, and recursively unset empty properties before using [json\_encode](http://php.net/manual/en/function.json-encode.php) to generate the response. Run the example below [here](https://www.tehplayground.com/yq8X04VLEF9ypc8Y) to see the result. ```php "https://openactive.io/", "@id" => "https://example.com/api/sessions/1402CBP20150217", "identifier" => "1402CBP20150217", "@type" => "SessionSeries", "organizer" => [ "@type" => "Organization", "name" => "Everyone Active", "url" => null, "logo" => [ "type" => "ImageObject", "url" => "", ], "email" => null, "telephone" => "01455 890508", "sameAs" => [ null, "" ], ], ]; echo json_encode_without_null_or_empty($testResponseStructure); function json_encode_without_null_or_empty($nestedarr) { $nestedarr = array_filter_recursive($nestedarr); return json_encode($nestedarr, JSON_PRETTY_PRINT); } function array_filter_recursive($array) { $clean = []; foreach ($array as $key => $value) { // First clean the array, in case it ends up empty if (is_array($value)) { $value = array_filter_recursive($value); // Remove arrays containing only a "type" property if (count($value) === 1 && isset($value["type"])) { continue; } } // ignore null values, empty strings, and empty lists if ($value === null || $value === "" || $value === []) { continue; } $clean[$key] = $value; } return $clean; } ``` # Testing RPDE feeds {% hint style="warning" %} Please note that all feeds must pass the **OpenActive Test Suite** in order to be usable by the largest number of data users. As described below, the interactive OpenActive Validator only validates the first 10 items / first 20 pages of a feed, whereas the OpenActive Test Suite runs the same checks over all items, in all pages, of all feeds. {% endhint %} ## OpenActive Validator The interactive online [OpenActive Validator](https://validator.openactive.io) allows the user to perform two types of validation: * [RPDE validation](https://validator.openactive.io/rpde): attempt to consume **only the** **first 20 RPDE pages** of an RPDE feed and check for common errors with the RPDE implementation * [JSON-LD validation](https://validator.openactive.io/): depending on the data supplied, check either (i) an individual item, (ii) **only the first 10 JSON-LD items** in a single feed page, or (iii) an Open Booking API request/response for validity The OpenActive Validator is useful during development to get instant feedback on the basic data structure and feed implementation, however it is not designed to comprehensively validate a complete implementation. ## OpenActive Test Suite The downloadable [OpenActive Test Suite](https://github.com/openactive/openactive-test-suite/) includes an option for comprehensively validating OpenActive open data feeds. Behind the scenes it runs the same checks as the [OpenActive Validator](https://validator.openactive.io/) for **every RPDE page** in the feed, and for **every JSON-LD item** in the feed. The OpenActive Test Suite produces a results page that allows the user to explore specific errors with individual data items, by loading those items into the interactive OpenActive Validator. ### Step 1: Install the test suite Clone the test suite repository locally, and install its dependencies. [Node.js](https://nodejs.org/en/download/) version 14 or above is required. ```bash git clone git@github.com:openactive/openactive-test-suite.git cd openactive-test-suite npm install ``` ### Step 2: Run validation Validation mode does not require any specific configuration, simply run the following command with the URL of the dataset site to be tested: ```bash npm run validate-feeds "http://reference-implementation.openactive.io/OpenActive" ``` ## Manual Testing Checklist Things to check to ensure the feed is implemented correctly. ### Most common implementation error Misreading the query in the specification is the single most common cause of incorrect implementation. Please read it carefully and ensure that brackets and comparators are used correctly. `>` not `>=` for example. Please ensure that you have implemented [this query](https://www.openactive.io/realtime-paged-data-exchange/#sql-query-example-for-timestamp-id) correctly: ```sql --include WHERE clause only if @afterTimestamp and @afterId provided WHERE (modified = @afterTimestamp AND id > @afterId) OR (modified > @afterTimestamp) ORDER BY modified, id ``` ### Feed integrity checks * Does the `next` url work as expected and return a valid page - it should never result in a 500 error. * "`modified`" must always be an integer, `afterTimestamp` / `afterChangeNumber` must also be an integer. * Does the `afterTimestamp` or `afterChangeNumber` of the next url always increase with each new page - if not the query has likely been badly implemented. * There should be "`deleted`" items in the feed. If these are missing, it is likely the feed has not been implemented correctly. * The `next` URL should be an absolute not relative URL. * Are all responses returned with header `Content-Type: application/json` * Check for duplicate IDs: items should not appear more than once in the feed if the source data is unchanging. * Pages should contain at least 500 items (this is a warning rather than an error). * Are the next URL parameters urlencoded? ### Last Page checks * Is the `next` URL present on the last page? The `next` URL on the last page should match the URL of the current page. * Check that the items array of the last page is empty To quickly access the last page: * **afterTimestamp**: If "modified" is an integer, put a high integer in for afterTimestamp to return the last page. (N.B. the spec currently allows for strings to be used for "modified", but a future spec will likely [make integers a must here](https://github.com/openactive/realtime-paged-data-exchange/issues/89)). * **afterChangeNumber**: Put a high integer in for afterChangeNumber to return last page ### Data quality checks * Does the feed include all historical data from the beginning of time and not just data in the future or from today's date? * Does the endpoint without any parameters return the first page (from the beginning of time)? * Does each page contain a "license" key? # Scaling RPDE feeds ## Approach to Scale A retention period should be applied to your data, so that sessions in the past are removed from the feed, while respecting the [RPDE invariants](https://www.w3.org/2017/08/realtime-paged-data-exchange/#core-concept). A CDN such as [CloudFlare](https://www.cloudflare.com/) is recommended to allow your RPDE endpoint to scale to millions of requests inbound. ## Retention Period In order to minimise the total number of items within of RPDE feeds, it is recommended that data publishers apply a retention period - especially for `Event,` `ScheduledSession` or `Slot` data items. The [Realtime Paged Data Exchange specification](https://www.w3.org/2017/08/realtime-paged-data-exchange/#deleted-items) specifies that: > If any record is added to the list or updated it must remain in the list in perpetuity while it is in an "updated" state, or remain in the list for at least 7 days from the point in time at which it transitioned to a "deleted" state {% hint style="info" %} The [high-volume proposal](https://github.com/openactive/realtime-paged-data-exchange/issues/93) for the RPDE specification is currently widely adopted, and hence it is recommended that `Slot` feeds that have a particularly high volume of small payload items wait 2 days before removing `"deleted"` items from the feed, in place of the specified 7 days. {% endhint %} ### Option 1: Retention period to minimise storage requirements If the objective of implementing a retention period is primarily to reduce the number of records stored, records representing events that occur in the past should be pruned by: * First setting their [`state` to the `"deleted"` state and updating the `modified` value](https://www.w3.org/2017/08/realtime-paged-data-exchange/#deleted-items). * Then after 7 days removing them from the feed. This may be implemented via a regular CRON job, for example. {% hint style="danger" %} Simply filtering out or removing opportunities from the feed that are in the past is not sufficient, because any opportunity that is edited to have a `startDate` in the past would then disappear from the feed without first transitioning into the [`"deleted"` state](https://www.w3.org/2017/08/realtime-paged-data-exchange/#deleted-items) with an updated `modified` value. Hence the previous version of such a record would live on forever in the downstream applications. {% endhint %} ### Option 2: Retention period to reduce feed size If the objective of implementing a retention period is primarily to reduce the size of the feed, an effective retention period can be implemented by having the first page of the feed start from the first relevant record (instead of from the beginning of time). This approach is useful for simple cases where a CRON job is not desirable. The approach can be implemented as follows: if the `@afterTimestamp` and `@afterId` parameters are **not** supplied to the RPDE endpoint (i.e. for the first page), use the query below to try to get the `@firstTimestamp` and `@firstId` and use these as default values: ```sql -- Execute ONLY if @afterTimestamp and @afterId NOT provided SELECT @firstTimestamp = modified, @firstId = id FROM ... WHERE startDate >= @now AND state <> "deleted" ORDER BY modified, id LIMIT 1 ``` For the first page only (where `@afterTimestamp` and `@afterId` parameters are **not** supplied), these default values are included within the `WHERE` clause of the RPDE query as below, with a slight change to the operands such that **`id >= @firstId`** to ensure the default value itself is included in the feed. For the first page, if no default values are returned, the RPDE query must exclude the `WHERE` clause as per the [specification](https://www.openactive.io/realtime-paged-data-exchange/#sql-query-example-for-timestamp-id), and return results from the beginning of time. Hence the query either uses: * For the first page: the default values supplied from the query above or otherwise returns results from the beginning of time * For all other pages: the values supplied by the parameters ```sql -- Include this WHERE clause only if @afterTimestamp and @afterId -- are NOT provided (first page), and if default values are available WHERE (modified = @firstTimestamp AND id >= @firstId) OR (modified > @firstTimestamp) -- Include this WHERE clause only if @afterTimestamp and @afterId -- are provided (not first page) WHERE (modified = @afterTimestamp AND id > @afterId) OR (modified > @afterTimestamp) -- If @afterTimestamp and @afterId not provided, and default values -- are not available, do not include WHERE clause ORDER BY modified, id ``` To further reduce the number of records in the feed, any record for an opportunity in the past can be rendered as `"deleted"` in the feed without any change to the `updated` value (note the record **must not** be removed from the feed). This means that records representing past opportunities are effectively frozen after they have occurred, and by default will live on forever in downstream applications. If the record is edited, the `modified` value must still be updated, at which point the record will be removed from downstream applications, to ensure historical accuracy. {% hint style="info" %} Alternative approaches for implementing this option are available in [this proposal](https://github.com/openactive/realtime-paged-data-exchange/issues/96). Feedback and thoughts very welcome. {% endhint %} ## CDN Configuration A CDN is simple to configure and requires a small amount of additional code within the RPDE endpoint. Note that in order for the CDN to be effective your application **should not** implement the [optional `limit` parameter specified in the RPDE specification](https://www.w3.org/2017/08/realtime-paged-data-exchange/#modified-timestamp-and-id). ### Basic Configuration The following CDN configuration options are recommended: * Configure the CDN to "Pass through and respect cache TTL headers" instead of overriding * In the application, vary the cache headers for RPDE pages as follows: * For all pages which contain greater than zero items, return a TTL of "60 minutes": * `Cache-Control: public, max-age=3600` * For the last page, which contains zero items, return a TTL of "8 seconds": * `Cache-Control: public, max-age=8` ### Worked Example The settings in the CDN Configuration section will create the behaviour described in this worked example automatically: #### Scenario * In this scenario, 200 data consumers are tracking the RPDE feed by polling at the end of the list (the last `next` URL). * Although each data consumer can choose a polling frequency arbitrarily, that frequency is not relevant to the calculations here, as it is the settings of the cache header that dictate the load on the origin server. It should also be noted that during normal operation the number of data consumers also does not impact the load on the origin server, and that 200 is used illustratively. * When the last page is requested, the first consumer would request the live page (creating one hit on the origin server), and the subsiquent 199 data consumers would receive a cached version. #### **"Sleep" mode** * For a feed whose source data is not being updated, the origin server receives one hit every 8 seconds and returns an empty list of `items` each time, and an identical `next` URL. * Hence the maximum load during "Sleep" mode is 8 requests/minute. #### "Live" mode * When an update to source data occurs, one of the 8-second interval requests will render a list of items and a new `next` URL. * The `items` list is rendered once by the origin server, and the subsiquent 199 data consumers would receive a cached version. * All 200 data consumers will follow the same next URL, and again the first request will be cached for the other 199 data consumers. * Hence the maximum load during "Live" mode is bounded by the response time of the "last" page, as the CDN will queue the requests from other data consumers waiting for this page. If the origin server is under high general load from other services, and the response time of the last page is increased, then the queue waits. This avoids a large number of data consumers adversely affecting the origin server performance during times of peak general load. #### Resyncs and rate limits If a data consumer decides to "resync" their data from the beginning of your feed, this will result in a temporary increase in request volume, as not all the pages earlier in the feed will still be cached in the CDN. This is not a frequent occurrence, however to ensure your system is protected from excessive requests, a rate limit behind your CDN of 10 requests/second is recommended. ### CloudFlare Walkthrough In order for [CloudFlare](https://www.cloudflare.com/) to respect your cache control headers, there are five simple steps to follow: #### 1) Set up CloudFlare as your DNS provider and proxy After you've [set up CloudFlare](https://support.cloudflare.com/hc/en-us/categories/200275218-Getting-Started) as your DNS provider, check requests are being routed through CloudFlare by enabling the orange cloud button. #### 2) Set up a page rule with a wildcard that covers your feeds Use the wildcards to ensure the rule covers all your feeds, for example: ``` *opendata.example.com/api/feeds/* ``` The page rule should have the following configuration: * **Cache Level:** Everything * **Origin Cache Control**: On * **SSL:** Flexible (if you do not have SSL configured on your own server) #### 3) Set Browser Cache Expiration to Respect Existing Headers On the Caching configuration page, ensure that following is set: * **Browser Cache Expiration:** Respect Existing Headers ![](/files/cmaqMQOugP3VjEXtMGV8) #### 4) Ensure that your feed does not inadvertently set any cookies Ensure that your web application or web server infrastructure does not set any cookies on the feed pages (for example load balancer [affinity cookies](https://azure.microsoft.com/en-gb/blog/disabling-arrs-instance-affinity-in-windows-azure-web-sites/)), as these will prevent CloudFlare from caching pages. #### 5) Test your configuration Inspect the headers returned by your page to see if CloudFlare is successfully caching your feed. A successfully cached page will return the following header: * **cf-cache-status: HIT** ### Further information The following articles will help you dive deeper into this in case you have any issues: * [Understanding and Configuring Cloudflare Page Rules (Page Rules Tutorial)](https://support.cloudflare.com/hc/en-us/articles/218411427-Understanding-and-Configuring-Cloudflare-Page-Rules-Page-Rules-Tutorial-) * [How Do I Tell Cloudflare What to Cache?](https://support.cloudflare.com/hc/en-us/articles/202775670-How-Do-I-Tell-Cloudflare-What-to-Cache-) * [Origin Cache-Control](https://support.cloudflare.com/hc/en-us/articles/115003206852-Origin-Cache-Control) * [What do the various Cloudflare cache responses (HIT, Expired, etc.) mean?](https://support.cloudflare.com/hc/en-us/articles/200168266-What-do-the-various-Cloudflare-cache-responses-HIT-Expired-etc-mean-) * [View HTTP headers in Chrome Dev Tools](https://developers.google.com/web/tools/chrome-devtools/network-performance/reference#headers) # Activity list references ## Using the OpenActive Activity List In order to allow your data to be easily searchable across a wide range of applications, it must contain references to the [OpenActive Activity List](https://www.openactive.io/activity-list/). The OpenActive Activity List is a standardised hierarchical list of physical activity types. It includes over 600 entries, each of which have a unique identifier in the form of a URL. A [JSON-LD definition](https://openactive.io/activity-list/activity-list.jsonld) of the OpenActive Activity List is available for live integration into applications, together with a [JavaScript library](https://www.npmjs.com/package/@openactive/skos) for navigating the list. Each opportunity within a booking or listing system must have an associated activity from the OpenActive Activity List. This is often achieved by providing a [dropdown list](#rendering-the-openactive-activity-list-with-skos-js) for the activity provider to select from when they are creating or updating an opportunity in the booking system. ### Including OpenActive Activity List references in your open data feed In the relevant [open data feed](/publishing-data/data-feeds/types-of-feed), the `@id` and `prefLabel` of at least one activity from the OpenActive Activity List must be included with each opportunity, along with an `inScheme` of `"https://openactive.io/activity-list"`, as shown below: ```javascript "activity": [ { "@type": "Concept", "@id": "https://openactive.io/activity-list#5e78bcbe-36db-425a-9064-bf96d09cc351", "prefLabel": "Bodypump™", "inScheme": "https://openactive.io/activity-list" } ] ``` See [here](https://validator.openactive.io/?url=https%3A%2F%2Fwww.openactive.io%2Fdata-models%2Fversions%2F2.x%2Fexamples%2Fsessionseries-split_example_1.json\&version=2.x) for a full feed example that references the OpenActive Activity List using the snippet above. {% hint style="warning" %} Please note that although the newer **`@id`** and **`@type`** are used here and throughout the rest of the OpenActive documentation and tooling, the OpenActive Activity List JSON-LD definition itself still uses **`id`** and **`type`** for backwards compatibility. {% endhint %} ### Multiple OpenActive Activity List references A booking system may choose to allow an opportunity to include multiple references to the OpenActive Activity List. If such a feature is implemented, a limit on the total number of references permitted per opportunity is recommended to discourage "tag spamming" (for example, a maximum of 3). An example of multiple references to the OpenActive Activity List is shown below: ```javascript "activity": [ { "@type": "Concept", "@id": "https://openactive.io/activity-list#eaa5a9bc-c23d-4643-80d9-8691646545be", "prefLabel": "Pool", "inScheme": "https://openactive.io/activity-list" }, { "@type": "Concept", "@id": "https://openactive.io/activity-list#ee7fa47c-617b-44ac-88fa-ef7b7ae03516", "prefLabel": "Snooker", "inScheme": "https://openactive.io/activity-list" } ] ``` ### Integration with an existing "activity types" controlled list If your booking system already has "activity types" available from a controlled list, these existing activity types should be mapped to the OpenActive Activity List. This is usually achieved by adding additional fields "OpenActive @id" and "OpenActive prefLabel" to your existing activity types table, and providing a [dropdown](#rendering-the-openactive-activity-list-with-skos-js) in your activity type editor. ### Loading the OpenActive Activity List within your application This [OpenActive Activity List JSON-LD definition](https://openactive.io/activity-list/activity-list.jsonld) **SHOULD** be retrieved at least nightly using an HTTP GET and cached within an application. This ensures that the most up-to-date version is displayed to the user, while also protecting against network failure when accessing the underlying resource. To access the JSON-LD definition the application **MUST** GET the URL `"https://openactive.io/activity-list/activity-list.jsonld"` which does not require a specific `Accept` header, and is cached via CDN. Note: there is no `www` in the URL. {% hint style="info" %} The JSON-LD definition is also available via a GET of the URL `"https://openactive.io/activity-list"` using an `Accept` header of `application/ld+json`, to conform with JSON-LD expectations, however this shorter URL **MUST NOT** be used in production. {% endhint %} ### Storing references to the OpenActive Activity List Within your application, it is advisable to store the full `@id` of an OpenActive Activity List `Concept` against each opportunity in your database, as the `prefLabel` and other properties are likely to change over time. Your application may also store the `prefLabel` alongside the `@id` at the point of the associating an OpenActive Activity List `Concept` with an opportunity, to remove the need to reference the activity list while outputting open data. It is the responsibility of the data user to use the latest `prefLabel` when rendering the open data it receives. ### Hardcoding OpenActive Activity List references If your booking system is restricted to a small number of different activities (e.g. [Run Together](https://data.runtogether.co.uk/) is restricted to just "Running"), it is usually better to hardcode the activity list references into your booking system. To find the **`@id`** simply [find the relevant activity in the OpenActive Activity List](https://activity-list.openactive.io/en/basic_find.html), then scroll down to the bottom the page to view a full example JSON-LD snippet for that specific activity, such as the screenshot below. This can be included in your open data feed. ![Screenshot of full example JSON-LD snippet from OpenActive Activity List website](/files/-M5m7k26wiyON2StpsgZ) ## Rendering the OpenActive Activity List with SKOS.js We recommend using [SKOS.js](https://www.npmjs.com/package/@openactive/skos) to implement any activity list client-side rendering, to allow your users to select an activity from a hierarchical representation of the OpenActive Activity List to associate with an opportunity. Maintaining the hierarchy and providing a typeahead search is important as with over 600 activities, and with many activities being more general terms, using an ordinary dropdown box becomes unwieldy for the user. ### Rendering a hierarchy from the JSON-LD definition of the OpenActive Activity List Although the use of [SKOS.js](https://www.npmjs.com/package/@openactive/skos) is recommended when reading the JSON-LD definition of the OpenActive Activity List in JavaScript, for other languages the JSON-LD may also be parsed directly. The [JSON-LD definition](https://openactive.io/activity-list/activity-list.jsonld) of the OpenActive Activity List includes the following key properties: | Property | Description | Include in open data feed | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | | `id` | The unique ID of the Concept, which should be stored and used when referencing the Concept. | Yes | | `prefLabel` | The primary display label for the Concept, in the English language. | Yes | | `altLabel` | The alternative display labels of the Concept, in the English language. When displaying the Concept to the user it is recommended that the array of `altLabel` be appended to the `prefLabel`, using the separator `" / "`. This logic does not apply to the `prefLabel` included in the open data feed. | No | | `topConceptOf` | Indicates that the Concept is at the top level of the hierarchy, when the value of this property is equal to `"https://openactive.io/activity-list"`. | No | | `broader` | An array of parent Concept IDs. Note this is a [polyhierarchical](https://en.wiktionary.org/wiki/polyhierarchy) list, and the same Concept may exist under multiple parents. | No | | `narrower` | An array of child Concept IDs. | No | To render a hierarchy: 1. Filter the data to return only Concepts with `topConceptOf` set to `"https://openactive.io/activity-list"`, to produce an initial list of top-level Concepts. 2. Recursively, for each Concept, lookup the `narrower` (child) Concept IDs. ### Activity List dropdown implementation example Below is a copy-and-pastable example of a searchable hierarchical dropdown that can be used within your booking system to allow the user to select an activity from the OpenActive Activity List. On load you may specify the **`@id`** to set the dropdown to an existing value. On selection the dropdown provides both **`@id`** and **`prefLabel`**, which can then be stored in your database and later used within your open data feed(s). The dropdown also provides an activity **`definition`** that may be displayed to the user for disambiguation. See the "**Result**" tab below for a live demo. {% embed url="" %} ### Our Parks: Activity List dropdown video demo An example of the above code in the Our Parks booking system is shown in the video below. Note "**OpenActive id**" and "**OpenActive prefLabel**" fields are displayed in the video for debugging purposes, but would ordinarily be hidden from the user. {% embed url="" %} ## Contribution links It is recommended that you provide links to contribute to the OpenActive Activity List, which encourage activity providers to participate in its maintenance and curation. Two different types of links are available: #### Contribute to the list For general contribution simply link to "[`https://openactive.io/activity-list/`](https://openactive.io/activity-list/)". This will take the user to a page where they can make suggestions about the activity list. Example: [Can't find an activity? Contribute to the list](https://openactive.io/activity-list/) #### Contribute to a specific activity The link to contribute to a specific activity is simply the URL of the **`@id`**. This will take the user to a page specific to the activity where they can make suggestions about the activity, including updating the definition of an activity. Example: [Contribute](https://openactive.io/activity-list#72d19892-5f55-4e9c-87b0-a5433baa49c8) ### Gladstone: Contribution links example An example of "Contribute" links within the Gladstone leisure management system is shown below. ![Screenshot of the "Contribute" links within the Gladstone system](/files/-M5lPxZfA3vlKXgwo7Tj) # Including geo coordinates ## Using geographic coordinates In order to maximise the usefulness of your data, it is highly recommended that it includes `GeoCoordinates`. Many data users will not use your data unless it includes a **`latitude`** and **`longitude`** within the `geo` property of each **`Place`**. ```javascript "location": { "@type": "Place", "url": "https://www.everyoneactive.com/centres/Middlesbrough-Sports-Village", "name": "Middlesbrough Sports Village", "description": "You can enjoy a huge range of different sports and activities at Middlesbrough Sports Village. There’s a state-of-the-art gym, a full programme of group exercise classes, indoor badminton sessions and a soft play area for kids. Outside, the Village boasts a full range of athletics facilities, including an eight-lane 400m track, a 10-lane 100m track, as well as four long jump pits. There’s also high jump, hammer throwing, javelin and pole vaulting facilities. The latest addition to the facility is a £1.6m, 250m velodrome, alongside five all-weather outdoor pitches and a skate park for BMX bikes, scooters and inline skaters.", "telephone": "01642 728555", "address": { "@type": "PostalAddress", "streetAddress": "Alan Peacock Way", "addressLocality": "Village East", "addressRegion": "Middlesbrough", "postalCode": "TS4 3AE", "addressCountry": "GB" }, "geo": { "@type": "GeoCoordinates", "latitude": 54.543964, "longitude": -1.20978500000001 } } ``` ## Licensing and republishing In order to publish geographic coordinates in your open data feed, you must ensure that the source of these coordinates (usually your geocoder) permits you to republish this data under an open licence. The [OpenCage Geocoder](https://opencagedata.com/) is an example of a geocoder that [permits such republishing](https://opencagedata.com/why-use-open-data). The basic [Google Geocoding API](https://developers.google.com/maps/documentation/geocoding/policies) licence **does not** permit open data republishing, and does not permit including the data in any externally facing API. ## Rendering a geocoding user interface Given the inaccuracy of geocoding for sports venues, we recommend using [leaflet.js](https://leafletjs.com/) (or a similar service) to implement a map to allow your users to move a pin to a specific location. ### Leaflet.js and OpenCage implementation example Below is a copy-and-pastable example of a drag-and-drop map pin and geocoder, that can be used within your booking system to allow the user to enter reliable latitude and longitude data.‌ It is powered by [OpenStreetMap](https://www.openstreetmap.org/), [open data compatible geocoding](https://opencagedata.com/), and [open source libraries](https://leafletjs.com/). On load you may specify the **`map.centerMap(lat, lng)`** to set the initial position of the map.‌ On pressing the geocode button, the geocoder provides both **`latitude`** and **`longitude`**, which can then be stored in your database and later used within your open data feed(s).‌ The map pin can also be used to adjust these values. See the "**Result**" tab below for a live demo. {% embed url="" %} ### Gladstone: Implementation example An example of the map pin interface within the Gladstone leisure management system is shown below. ![Screenshot of the drag-and-drop map pin within the Gladstone system](/files/-M6_orpcphKcApkwBs7q) # Schedules A [`Schedule`](/data-model/types/schedule) is a representation of a recurrence rule, compatible with the [iCalendar specification](https://tools.ietf.org/html/rfc5545), designed to be extrapolated by the data user into individual occurrences. A [`PartialSchedule`](/data-model/types/partialschedule), by contrast, **must** **not** be extrapolated by the data user, and is for information only. ## Outputting a `Schedule` to a feed A [`Schedule`](/data-model/types/schedule) **must** be included in the `SessionSeries` feed for systems that display occurrences of a sessions directly from a recurrence rule (as opposed to first materialising them into a database). ### Complementary `ScheduledSession`s It is recommended that a `ScheduledSession` feed be published alongside any `SessionSeries` feed that contains `Schedule`s, and that this `ScheduledSession` feed also includes `remainingAttendeeCapacity`. If implementing the Open Booking API, such a complementary feed is **required**. When `ScheduledSession`s are included in a feed or `subEvent`, they must not include all occurrences that are generated from the recurrence rule, and instead **must only** include those events that either: 1. Have bookings associated (i.e. where `remainingAttendeeCapacity` < `maximumAttendeeCapacity`) 2. Are an exception to the recurrence rule defined in the `Schedule`. 3. Have any other properties that differ from those defined in `SessionSeries`. Such events would need to have been materialised in a database within the booking system already, so **under no circumstances should new occurrence records be generated for the sole purpose of outputting them to an OpenActive `ScheduledSession`s feed**. This constraint is necessary to prevent recurrence rules from creating a high volume of redundant data in its `ScheduledSession` feed. For example: if a `SessionSeries` has a recurrence rule for a weekly event with an `endDate` in 2050, only the `ScheduledSession`s that have been booked at least once or have been manually edited would appear in the `ScheduledSession` feed. ### Using templates When such `ScheduledSession`s are included, the `Schedule` **must** also include an `idTemplate`, such as the below, which matches the pattern of the `@id` of the `ScheduledSession`s (noting the `startDate` placeholder **must** use a string format of `YYYY-MM-DDThh:mm:ssZ` (e.g. `1997-07-16T19:20:00Z`): ```javascript "idTemplate": "https://api.example.org/session-series/123/{startDate}" ``` A `urlTemplate` may also be included in the `Schedule` using the same `startDate` placeholder. Note that if a single `ScheduledSession` that was previously generated by a `Schedule` is rescheduled to a different start time, its original `@id` **must** be retained (which contains the original start time), to ensure that it still hides the same generated occurrence (see below), and to ensure that [Change of Logistics Notifications](https://openactive.io/open-booking-api/EditorsDraft/#change-of-logistics-notifications) are still triggered. Also note that if a `Schedule` is updated, it must include `exceptDates` for any overlaps with any explicitly defined occurrences created from previous `Schedule`s, as they might have been rescheduled based on a previous `@id`. ## Processing a `Schedule` found in a feed In order to process a [`Schedule`](/data-model/types/schedule) together with complimentary `ScheduledSession`s, the data user must do the following: 1. Generate all occurrences from a `Schedule` in the future, taking into account the `exceptDate` property. * Take the `scheduledEventType`, property and use it for the `@type` property of each occurrence. * Use an [RRULE](https://icalendar.org/iCalendar-RFC-5545/3-8-5-3-recurrence-rule.html) library to calculate the `startDate` of the occurrences based on the contents of the `Schedule`. `DTSTART` must be determined by using the `startDate`, `startTime`, and `scheduleTimezone` of the `Schedule` together, for example: * ``` { "@type": "Schedule", "startDate": "1997-09-02", "startTime": "09:00", "endTime": "10:00", "duration": "PT1H", "scheduleTimezone": "America/New_York", "repeatFrequency": "P1D", "repeatCount": 10 } ``` ``` DTSTART;TZID=America/New_York:19970902T090000 RRULE:FREQ=DAILY;COUNT=10 ``` * For the avoidance of doubt: the `startDate` and `startTime` of the `Schedule` are in "local time" based on the `scheduleTimezone`. * Use the `duration` of the `Schedule` to calculate the `endDate` of each occurrence. * Render the calculated `startDate` and `endDate` for each occurrence to UTC using a string format of `YYYY-MM-DDThh:mm:ssZ` (e.g. `1997-07-16T19:20:00Z`) for placeholder replacement. * Take the `idTemplate` property (if provided) and substitute the `startDate` placeholder with the calculated string value of `startDate` (and do the same with `endDate`). Use the resulting string as the value of the `@id` property for the occurrence. * Take the `urlTemplate` property (if provided) and substitute the `startDate` placeholder with the calculated string value of `startDate` (and do the same with `endDate`). Use the resulting string as the value of the `url` property for the occurrence. 2. To account for any changes in the `Schedule` since the last time it was updated, store the generated occurrences as follows: 1. Upsert all generated occurrences, marking them with the `modified` RPDE timestamp associated with the `Schedule`. Ensure that any explicitly defined occurrences that may have been generated from a previous run of Step 3 (below) are not overwritten by this step. 2. Delete all generated occurrences that do not have an older modified RPDE timestamp associated with the Schedule. 3. Ensure the generated occurrences are hidden by any matching explicitly defined occurrences (e.g.`ScheduledSession`s found in a `subEvent` or `ScheduledSession` feed), using the explicitly defined occurrence in its entirety based on its `@id`. ## Example Extract from `SessionSeries` feed: ```javascript { "@type": "SessionSeries", ... "eventSchedule": [ { "@type": "Schedule", "repeatFrequency": "P1W", "startDate": "2018-03-01", "endDate": "2018-03-29", "startTime": "08:30", "endTime": "09:30", "byDay": [ "https://schema.org/Thursday" ], "duration": "PT1H", "exceptDate": [ "2018-03-15T08:30:00Z", ], "scheduleTimezone": "Europe/London", "scheduledEventType": "ScheduledSession", "idTemplate": "https://api.example.org/session-series/1402CBP20150217/{startDate}", "urlTemplate": "https://example.org/session-series/1402CBP20150217/{startDate}" } ] } ``` Extract from `ScheduledSession` feed: ```javascript { "state": "updated", "kind": "ScheduledSession", "id": "C5EE1E55-2DE6-44F7-A865-42F268A82C63", "modified": 1521565719, "data": { "@context": "https://openactive.io/", "@type": "ScheduledSession", "@id": "https://api.example.org/session-series/1402CBP20150217/2018-03-15T10:30:00Z", "identifier": "C5EE1E55-2DE6-44F7-A865-42F268A82C63", "superEvent": "https://example.com/api/session-series/1402CBP20150217", "startDate": "2018-03-15T10:30:00Z", "endDate": "2018-03-15T11:30:00Z", "duration": "PT1H", "eventStatus": "https://schema.org/EventScheduled", "maximumAttendeeCapacity": 10, "remainingAttendeeCapacity": 10, "url": "https://example.org/session-series/1402CBP20150217/2018-03-15T10:30:00Z" } }, { "state": "updated", "kind": "ScheduledSession", "id": "C5EE1E55-2DE6-44F7-A865-42F268A82C64", "modified": 1521565719, "data": { "@context": "https://openactive.io/", "@type": "ScheduledSession", "@id": "https://api.example.org/session-series/1402CBP20150217/2018-03-22T08:30:00Z", "identifier": "C5EE1E55-2DE6-44F7-A865-42F268A82C64", "superEvent": "https://example.com/api/session-series/1402CBP20150217", "startDate": "2018-03-22T08:30:00Z", "endDate": "2018-03-22T09:30:00Z", "duration": "PT1H", "eventStatus": "https://schema.org/EventScheduled", "maximumAttendeeCapacity": 10, "remainingAttendeeCapacity": 3, "url": "https://example.org/session-series/1402CBP20150217/2018-03-22T08:30:00Z" } } ``` Illustration of resulting opportunities presented to the end user: ![](/files/-M8LqjS5ZagZe7W_zwr8) # Dataset Sites Dataset Sites help data users to find your data, and interact with you if they find any issues. They are both human and machine readable, and allow your data to appear everywhere from the [OpenActive Status Page](https://status.openactive.io) to the [Google Dataset Search](https://toolbox.google.com/datasetsearch/search?query=openactive). Example dataset sites: [GLL](http://data.better.org.uk), [Fusion Lifestyle](https://opendata.fusion-lifestyle.com/OpenActive) ## Overview * In order for data users to find your data feeds, and for your dataset to be featured on the [OpenActive status page](https://status.openactive.io/), you must deploy the [Dataset Site Template](/publishing-data/dataset-sites#step-1-build-dataset-sites-into-your-system) (a simple mustache template) within your codebase. * For booking systems or bespoke websites with a **single database** and one set of OpenActive data feeds, a single Dataset Site is likely to be sufficient for your organisation. This can be achieved by simply hard-coding the JSON passed into the mustache template. * If you are a booking system with **multiple databases**, each of which has their own set of OpenActive data feeds, a Dataset Site is required for each customer. This can be achieved using customer configuration to drive the mustache template. * You need to create a [GitHub issues board](/publishing-data/dataset-sites#step-2-github-issues-board-creation) for each Dataset Site. * If you are publishing multiple dataset sites you also need to provide a [Data Catalog](/publishing-data/dataset-sites#step-5-providing-a-data-catalog-multiple-databases-only). * Data publishers should be encouraged to provide links to their Dataset Site from their own website. ### What is a Dataset Site? * A web page that can be referenced when discussing the dataset. * A human and machine readable licence associated with the data (the Dataset Site contains invisible metadata which allows its details to be read automatically). * A human and machine readable rights statement to specify how dataset users (innovators who want to build on top of/use your data) should attribute your data. * An accessible "single point of truth" that explains where the data can be found. * Links to documentation relating to the format of the data, including the specifications it follows, and the data fields it contains. * A place where the community can contribute with comments, and raise issues - all Dataset Sites are linked to a GitHub issues board (e.g. [this one](https://github.com/gll-better/opendata/issues)) that allows data users to raise issues in the open. {% hint style="warning" %} A machine-readable dataset site is essential when publishing open data, and every dataset published within the OpenActive community to date has had one. However, the specification that describes a standard OpenActive dataset site is still yet to be formally defined, and has instead evolved as a de facto standard. As such, this documentation is still based on a [draft model](/data-model/types/dataset) that is designed to inform the OpenActive specification work with implementation feedback. It is mostly stable and has been largely unchanged for 2 years. However, it is still **subject to change**, as the [Dataset API Discovery specification](https://www.openactive.io/dataset-api-discovery/EditorsDraft/) is yet to be formally released, and feedback is very welcome, both within the relevant OpenActive [repository](https://github.com/openactive/dataset-api-discovery/issues/) and on the related schema.org [PR](https://github.com/schemaorg/schemaorg/pull/2635) and [issue](https://github.com/schemaorg/schemaorg/issues/1423). To minimise any uplift work required to conform to the formal specification when it is released, it is **recommended** that you use [one of the libraries below](/publishing-data/dataset-sites#net-php-and-ruby-libraries) where possible. These libraries will be updated to meet the latest specification, and when used in their simplest mode ([RenderSimpleDatasetSite](https://github.com/openactive/OpenActive.DatasetSite.NET/#simple-implementation), [renderSimpleDatasetSite](https://packagist.org/packages/openactive/dataset-site#user-content-rendersimpledatasetsitesettings-supportedfeedtypes) or [TemplateRenderer.new](https://www.rubydoc.info/gems/openactive-dataset_site/0.1.1#Usage)) will only require a simple dependency update from you to do so. {% endhint %} ## Step 1: Build Dataset Sites into your system The Dataset Site Template is very easy to use and quick to apply - it's essentially one of two mustache templates and an associated JSON structure. It is designed to work with minimal effort with an extremely [wide range of platforms and languages](https://mustache.github.io/). It works across all browsers, and includes fully compliant DCAT and schema.org machine-readable metadata to ensure it is compatible with [Google Dataset Search](https://toolbox.google.com/datasetsearch/search?query=openactive). The[ dataset site template repository](https://github.com/openactive/dataset-site-template) contains two mustache template options for creating an OpenActive dataset site. ### Template hosting options There are two templates available, depending on your use case. #### Option 1: Single-file template The [Dataset Site Single-file Template](https://openactive.io/dataset-site-template/datasetsite.mustache) is a self-contained mustache template of an HTML page that contains embedded CSS, two embedded images, and fonts loaded via the CDNs of [Google Fonts](https://fonts.google.com/specimen/Source+Sans+Pro) and [cdnjs (powered by Cloudflare)](https://cdnjs.com/libraries/font-awesome/4.7.0). 1. Use one of the options below to dynamically render the 'single-file template' and output the result at an endpoint, for example `https://example.com/openactive/`. #### Option 2: CSP compatible template with separate self-hosted static assets The [Dataset Site CSP Compatible Template](https://openactive.io/dataset-site-template/datasetsite-csp.mustache) is a mustache template of an HTML page that references [self-hosted static assets](https://openactive.io/dataset-site-template/datasetsite-csp.static.zip). This template must be rendered using a reference to its stylesheet at its self-hosted location. This is useful for implementations that have a [Content Security Policy (CSP)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) in place. 1. Host the [CSP compatible static assets](https://openactive.io/dataset-site-template/datasetsite-csp.static.zip) somewhere on the same domain as your dataset site. 2. Use one of the options below to dynamically render the 'CSP compatible template' ensuring that the `"staticAssetsPathUrl"` references the location of your self-hosted assets, without a trailing slash (`/`) (this can be a relative or absolute URL). 3. Output the result at an endpoint, for example `https://example.com/openactive/`. ### .NET, PHP, Ruby and JavaScript/TypeScript Libraries Several libraries are available that make it really easy to render either dataset site template, accepting basic settings to configure your dataset site automatically. The table below lists the available OpenActive libraries: | Language | Open Opportunity Data Feeds | Dataset Site | | ----------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | .NET | [OpenActive.NET](https://www.nuget.org/packages/OpenActive.NET/) | [OpenActive.DatasetSite.NET](https://www.nuget.org/packages/OpenActive.DatasetSite.NET/) | | PHP | [openactive/models](https://packagist.org/packages/openactive/models) | [openactive/dataset-site](https://packagist.org/packages/openactive/dataset-site) | | Ruby | [openactive](https://rubygems.org/gems/openactive) | [openactive-dataset\_site](https://rubygems.org/gems/openactive-dataset_site) | | JavaScript / TypeScript | [@openactive/models-ts](https://www.npmjs.com/package/@openactive/models-ts) | [@openactive/dataset-site-template](https://www.npmjs.com/package/@openactive/dataset-site-template) | | CLI | - | [npx @openative/dataset-site-template](https://www.npmjs.com/package/@openactive/dataset-site-template#cli-static-dataset-site-generator) | ### Other Languages A basic example of following the below render steps can be found [here](https://github.com/openactive/dataset-site-template-example-dotnet/blob/master/DatasetSiteTemplateExample/Program.cs), and can be readily ported into other languages. An explanation of how this works is included below. The [Dataset Site Template](https://github.com/openactive/dataset-site-template/) is one of two mustache templates of an HTML page. It works across all browsers, and includes fully compliant DCAT and schema.org machine-readable metadata to ensure it is compatible with [Google Dataset Search](https://toolbox.google.com/datasetsearch/search?query=openactive). Steps to render the template: 1. Construct the JSON-LD to match the format found in [this example](https://validator.openactive.io/?url=https%3A%2F%2Fopenactive.io%2Fdataset-site-template%2Fexample.jsonld\&version=2.x\&validationMode=DatasetSite), following [this documentation](/data-model/types/dataset). 2. Find a [mustache library](https://mustache.github.io/) for your platform or language. 3. Write code to do the following: * Stringify the input JSON, and place the contents of the string within the `"jsonld"` property at the root of the JSON itself (i.e. serialised JSON embedded in the original deserialised object). This is important as it is used to populate the machine-readable `