Implementing RPDE
Last updated
Last updated
Always use a JSON library to generate the output, and never construct the JSON manually.
Language-specific examples of library implementations that also support the removal of nulls, empty strings and empty arrays - as the specification stipulates - are included here.
Several libraries are available that make it really easy to create open opportunity data feeds.
The table below lists the available OpenActive libraries:
When concurrent transactions are used to write to tables that power RPDE feeds, the timestamp and change number update must be done outside of the transaction, with high accuracy timestamp, and a "visible" timestamp column added.
Delayed item interleaving is a race condition that occurs when two concurrent transactions containing "timestamps" or "change numbers" are committed out of order. For example using the Modified Timestamp and ID ordering strategy:
Transaction 1: Starts at 10:00:01 and updates RPDE item timestamps
Transaction 2: Starts at 10:00:02 and updates RPDE item timestamps
Transaction 2: Commits
Data consumer reads the feed at 10:00:03, and gets the latest items up to 10:00:02
Transaction 1: Commits
Items from Transaction 1 appear in the feed with timestamp 10:00:01, but the data consumer has already moved past them
The same issue can be demonstrated with the Incrementing Unique Change Number ordering strategy:
Transaction 1: Creates change number 001 and updates RPDE item timestamp
Transaction 2: Creates change number 002 and updates RPDE item timestamp
Transaction 2: Commits
Data consumer reads the feed up to 002
Transaction 1: Commits
Item from Transaction 1 appears in the feed with change number 001, but the data consumer has already moved past it
In order to prevent this race condition, one solution is to simply separate the more intensive work of the transaction from the atomic timestamp and change number update, filter out recent changes in the feed, and ensure you are using accurate timestamps:
First commit the transaction, then update the timestamps or change numbers after the transaction has been committed as an atomic operation, outside of a transaction, using GETDATE()
or similar
Ensure the RPDE endpoint filters out all items with a "modified" date after 2 seconds in the past, to delay items appearing in the feed
If using the Modified Timestamp and ID ordering strategy, use a timestamp column with a high degree of accuracy (e.g. datetime2
in SQL Server).
Updating items in the feed without updating their timestamp/change number immediately does not have any negative effects, as data consumers who are reading the feed will read the updated item earlier than they would otherwise instead of an older version, then read it again after the timestamp/change number is updated as they would normally.
Using the RPDE endpoint filter to delay items appearing is a belt-and-braces measure that ensures that small variances between timestamp and change number update order are accounted for under high database load, by only presenting data to the data consumer after a small delay, when all timestamp/change number updates have commited.
We highly recommend using the OpenActive.NET library for .NET implementations. It includes strongly typed classes for the OpenActive data model including beta properties, and fully compliant serialisation methods too.
Use JSON.NET / Newtonsoft.Json with a ContractResolver
to remove empty strings, and NullValueHandling
to remove nulls. Empty list values must be removed by setting them to null in the generating code, aided by the extension ToListOrNullIfEmpty
which is useful at the end of LINQ expressions.
We highly recommend using the OpenActive PHP Models Library for PHP implementations. It includes strongly typed classes for the OpenActive data model including beta properties, and fully compliant serialisation methods too.
The specification requires that no null or empty strings are present in the OpenActive feed. To achieve this, define the RPDE response structure as nested arrays, and recursively unset empty properties before using json_encode to generate the response.
Run the example below here to see the result.
Language
Open Opportunity Data Feeds
Dataset Site
.NET
PHP
Ruby
JavaScript / TypeScript