SCADA Data Gaps: What Missing Timestamps Do to Your Reported Numbers

Every performance number you publish carries an invisible asterisk: it assumes the data underneath it is complete. It almost never is. A renewable plant generates a relentless stream of time-series data — irradiance, power, temperatures, statuses, setpoints, thousands of tags sampled every few seconds and pushed through a SCADA system into a historian. Somewhere between the sensor and the monthly report, some of that stream goes missing. A radio link drops for twenty minutes. A logger reboots. A tag gets renamed during a firmware update and the old name stops recording. None of it is dramatic. All of it changes your numbers. The danger is not the gap itself. It is that most reporting systems paper over gaps automatically, so the person reading the report never learns the data was incomplete — and quietly trusts a number built on a partial record.

Six ordinary ways time-series data goes missing

Gaps are not a single phenomenon. They arrive through at least six distinct routes, and because each behaves differently, each damages your numbers differently.

• •Communications dropouts. A network or radio link fails and a block of intervals simply does not arrive. The gap is usually obvious — a clean hole in the record — but its effect depends entirely on what was happening during the hole.
• •Logger or device reboots. A device restarts and loses the intervals during startup. These gaps are short, frequent, and easy to dismiss individually while adding up to something material across a fleet.
• •Tag renames and remaps. During a firmware update or reconfiguration, a signal's tag name changes. The historian keeps recording under the new name and stops under the old one. Anything querying the old tag now sees a permanent gap, and anything summing across the boundary double-counts or under-counts.
• •Historian downsampling. To save storage, historians compress older data — replacing high-resolution samples with averages over longer windows. The data is not missing exactly, but its resolution is, and a five-minute event is invisible in hour-averaged history.
• •Clock and time-zone errors. A device on the wrong time zone, or a daylight-saving shift handled inconsistently, misaligns data against irradiance and against other devices. The data is all present — it is simply attributed to the wrong moment.
• •Quality-flag stripping. Good SCADA systems tag suspect values with a quality flag. When data is exported to a reporting tool that ignores the flag, bad values are treated as good, which is worse than a clean gap because nothing signals the problem.

Why interpolation hides the problem instead of solving it

The default response to a gap is to fill it — draw a straight line between the last good value and the next one, or carry the last value forward. This is convenient, and for some purposes defensible. But interpolation makes a specific, often wrong, assumption: that nothing interesting happened during the gap. Consider a communications dropout during a cloud transient. The power before the gap is high, the power after is high, and linear interpolation fills the gap with high power — erasing the dip that actually occurred. Your energy total is now overstated, and the availability calculation records the plant as producing when it may have been curtailed or tripped. The gap has not been handled; it has been hidden, and hidden in the direction that flatters the number. The deeper issue is that interpolated data looks identical to real data downstream. Once the straight line is drawn, no report distinguishes the invented values from the measured ones. The uncertainty has been erased from the record precisely where it mattered.

What each gap type does to a specific KPI

Different KPIs are vulnerable to different gaps, which is why a single 'data is fine' assurance is meaningless. The table below maps the mechanism.

Detecting gaps your dashboard will not show you

The reason gaps persist is that the tools built to display data are not built to audit its completeness. A dashboard shows you what is there; it says nothing about what is missing. Catching gaps requires looking at the record deliberately.

1. 1.Count expected versus received intervals. The simplest and most powerful check: for any tag over any period, you know how many intervals should exist — the period divided by the sample rate. Compare that to how many non-null, good-quality values you actually have. The ratio is your completeness for that tag. Anything below 100 per cent is a gap you did not know about.
2. 2.Watch the boundaries, not just the holes. Tag renames and clock errors do not produce holes — they produce misattribution. Detecting them means checking continuity across configuration changes and verifying that each device's timestamps align with a known reference like solar noon or a co-located sensor.
3. 3.Respect the quality flags. If your SCADA system produces quality flags, any reporting pipeline that discards them is throwing away the one signal designed to catch bad data. Carrying the flag all the way through to the report is unglamorous and essential.

A completeness metric worth putting in the monthly report

Here is the single change that does the most good: report data completeness as a number, next to the KPIs it underpins. 'Performance ratio 82.4 per cent, computed over 98.7 per cent complete data' tells the reader something 'Performance ratio 82.4 per cent' never could — it says how much to trust the figure. This does two things. It makes incompleteness visible, so a month with poor data quality cannot masquerade as a month with poor performance, or vice versa. And it creates an incentive to fix the underlying data pipeline, because completeness is now a reported number that people notice when it drops.

Handling historical gaps honestly

When you discover that a stretch of history was computed over incomplete data, the temptation is to interpolate retroactively and move on. The more defensible path depends on what the number was used for. For internal trending, recompute the affected KPIs with an explicit completeness figure attached, and flag any period where completeness was low enough to make the number unreliable. For any figure that fed a settlement, a guarantee test or an external report, the same principle applies as with any data correction: if the gap materially changed a number someone relied on, note it with a dated correction rather than silently restating it. The record of what you knew, and when, is itself part of the asset's integrity.

Frequently Asked Questions

What is an acceptable data completeness threshold for solar reporting?

There is no single universal threshold, but the operational principle is that completeness should be measured and reported rather than assumed. Many operators treat high-90s per cent completeness as the working target for KPI-grade data, and flag any reporting period that falls below their internal threshold as provisional. The key point is that completeness is a number to publish alongside the KPI, not a box to tick privately — the reader of the report needs to know how complete the underlying data was.

Does interpolated data invalidate an availability claim?

It can, depending on where the interpolation falls. Interpolating across a communications gap that coincided with a fault or curtailment fills the gap with assumed production, which overstates availability by erasing the very event availability is meant to capture. Interpolation across genuinely uneventful intervals is less harmful. The problem is that once data is interpolated, downstream reports cannot tell invented values from measured ones — so the honest approach is to track which intervals were interpolated and disclose it.

How do daylight saving shifts corrupt time-series data?

A daylight-saving transition, handled inconsistently across devices, shifts a plant's data by an hour relative to its own irradiance reference and relative to other devices. Because all the data is present — merely mis-timestamped — the corruption is invisible to any check that only looks for missing values. It shows up as a sudden apparent drop in performance ratio or a phase shift against solar noon around the transition dates. Consistent time-zone handling, ideally in UTC internally, prevents it.

What happens to KPIs when a tag is renamed mid-period?

A tag rename creates a permanent gap under the old name and a fresh record under the new one. Any KPI that queries the old tag sees the plant stop reporting; any KPI that sums across the rename boundary either under-counts (missing the new-name data) or double-counts (if both names briefly recorded). Because the plant itself never changed, the error is purely an artefact of configuration management — which is why rename events should be logged and reconciled deliberately, not discovered later as an unexplained discontinuity.

The takeaway

Complete data is an assumption, and unexamined assumptions are where reporting quietly goes wrong. You cannot eliminate gaps, but you can measure them, report them, and stop letting interpolation hide them. A completeness figure beside every KPI is the cheapest credibility you will ever buy — and the foundation for any analysis that claims to reconcile expected generation against actual.

Sources & Further Reading