Power plant schedules under SOGL: a data platform for transmission system operation

Starting point

Under the EU System Operation Guideline SOGL (EU 2017/1485), power plant operators report their dispatch planning: per unit and day, 96 quarter-hourly values for generation, balancing reserves, redispatch and availability. At project start, around 120,000 files and 26 GB arrived every day, set to reach 2.1 million files within a year. Every file must be validated, acknowledged to its sender and reliably passed on to forecasting and grid operation.

Solution

We built an event-driven platform on Kubernetes in Azure. .NET services import the files, and a dedicated validation service checks them via Kafka against XSD and business rules at document, time series and point level. The sender receives an IEC 62325 acknowledgement that can reject individual time series or the whole message with a reason code. Valid data is stored in PostgreSQL and distributed to downstream systems via Kafka topics and file exports.

Scaling to seventeen times the volume

• Duplicates: 92% of time series are re-sent unchanged, only with a new version number. A hash of the 96 values identifies them, so they are not stored again. At target volume this reduces the write load on paper from 105,000 to around 8,400 inserts per second, and storage from 1.36 TB to around 110 GB a day.
• Persistence: The load test located the bottleneck in row-by-row writes to a single database. We switched to bulk inserts, a time series database with compression and fast hash lookups.
• Silent failures: Very large validation responses were lost in the message broker without an error, and the affected files stayed in “validation pending”. We aligned message sizes across the entire chain, and large documents now pass through in full.
• Test data: Unit-level schedules count as inside information under REMIT. Load tests therefore run on synthetic files without duplicates, which represents the worst case.
• Forecasting: A new, lean exchange format based on Protobuf delivers one package per unit and business type, with no master data on the wire.

Result

The platform validates and acknowledges every file automatically, across five environments up to production. In the load test, 80,000 files went through in just over an hour at a success rate of around 99%, without any benefit from deduplication. Bottlenecks and savings are backed by numbers, and the architecture is designed for seventeen times the volume.