Every model on the NatGas Edge platform is documented end to end — the data sources, the target construction, the training procedure, the validation regime, and the known limitations. Credibility comes from the audit trail, not the marketing. Select a methodology below to read the full walkthrough.
This methodology explains, end to end, how the Renewables signal on the NatGas Edge platform is produced. It begins with the raw data sources, walks through the construction of the absorption ratio as an analytical target, describes the weather calibration model that produces physics potential, details the machine learning model that forecasts forward absorption, and explains how the final outputs — forecast gas displacement in Bcf — are derived. It is written for readers who want to understand the system rigorously, including the modelling decisions, the trade-offs taken, and the limitations.
It is not investment advice. Every output described here is the product of statistical modelling of historical data, with all the limitations that implies.
Most public renewable forecasts stop at one number: how much electricity wind and solar are expected to generate. That number, on its own, is not what determines natural gas prices.
What matters for gas is how much of that generated electricity actually displaces gas-fired generation. In high-renewable seasons, US grid operators routinely curtail solar and wind output — instructing inverters to reduce production — because gas plants cannot ramp below minimum generation thresholds, transmission corridors saturate between generation zones and load centres, or demand simply isn't high enough to clear the supply. A sunny May afternoon in California can see 30% of available solar curtailed, even with the panels physically capable of generating more. The energy is available, the demand exists somewhere on the grid, but the system constraints prevent the supply from reaching the load.
The metric that captures this is the absorption ratio: the fraction of physically available renewable output that the grid actually dispatches and uses. Multiplied by physics potential, it gives genuine forecast generation. Converted via heat rate, it gives forecast gas displacement in Bcf — which is what moves nat gas prices.
The foundation of the renewables pipeline is the US Energy Information Administration's Form 930 dataset, which publishes hourly electricity operations data for every US balancing authority. For each hour, EIA-930 reports net generation by fuel type — including hourly TWh produced by wind and solar — for the major ISOs and balancing authorities we cover: CAISO (California), ERCOT (Texas), MISO (Midcontinent), SPP (Southwest Power Pool), PJM (Mid-Atlantic), NYISO (New York), and ISO-NE (New England).
The data is ingested daily into our pipeline with a typical lag of 1–2 days between operational date and EIA publication. Seven years of historical hourly data are maintained, allowing us to model regime behaviour across the rapid build-out of renewables that has occurred since 2018. EIA-930 also publishes day-ahead demand forecasts produced by each balancing authority, which we use as one of the forecast inputs.
To compute what wind and solar could produce under given conditions — the physics potential — we need to know how much capacity exists in each ISO at each point in time. EIA Form 860 publishes generator-level data for every utility-scale plant in the United States, including nameplate capacity, online date, retirement date, and location. We aggregate this to ISO × fueltype monthly capacity series, accounting for the continuous fleet evolution.
This is critical because the renewables fleet has grown substantially across our seven-year history. A 5 TWh generation hour in 2019 represents a very different absorption ratio than the same generation in 2026 against a much larger installed base. Capacity-aware modelling separates the regime shift from the absorption behaviour.
Weather is the primary driver of renewable physics potential — solar irradiance for solar, wind speed for wind. We ingest forecasts and actuals for the five major US natural gas demand regions: South Central, East, Midwest, Mountain, and Pacific. Each region is anchored to the major population centre within it, which provides a good proxy for demand-side weather (heating and cooling degree days follow population, and gas demand follows degree days).
For each region we maintain three forecast horizons — near (1–5 day), mid (6–10 day), and far (11–15 day) — updated daily. This gives us forecast weather at the granularity the model needs to predict forward absorption regimes, plus actuals to compute historical anomalies against five-year seasonal norms.
The forecast model also incorporates:
Before we can model absorption forward, we need to know what the grid could have produced under each historical weather condition — the physics potential. This is computed daily per ISO × fueltype using a calibration model that learns the empirical relationship between weather inputs and actual generation, calibrated against EIA-930 actuals.
A pure physics model — Beer-Lambert irradiance equations for solar, cube-of-wind-speed power curves for wind — would compute potential output from first principles given perfect weather data at every panel and turbine. We do not attempt this. The compute and data requirements (hourly irradiance and wind speed at every fleet location for seven years) are prohibitive, and the calibration error from gaps in fleet location data would dominate.
Instead, we calibrate empirically. For each ISO × fueltype, we fit a model that learns: given a weather signal in this region on this day of the year with this much installed capacity, what generation would we have expected absent curtailment? The model is fit only on uncurtailed days (days where actuals match what physical conditions should have produced), giving us a clean read on the relationship between weather, capacity, and unconstrained output.
The calibration produces two key coefficients per ISO × fueltype:
The use of the 90th percentile (rather than the mean or maximum) is deliberate. The mean would understate fleet capability by averaging in low-performance days. The maximum would overstate it by anchoring to outlier days that may not represent reproducible behaviour. The 90th percentile captures consistent high-performance — what the fleet does on a good day — which is the right benchmark for physics potential as an absorption denominator.
For each daily row per ISO × fueltype, the calibration model produces:
These are computed by applying β_p90 to the forward weather forecast at each horizon. The absorption ratio is then defined as actual_twh / physics_potential_twh for historical days, and modelled forward via the ML pipeline described below.
For historical days, we compute the absorption ratio directly:
Values close to 1.0 mean the grid absorbed almost all available output — minimal curtailment. Values significantly below 1.0 mean curtailment was material, whether due to demand insufficiency, minimum gas generation requirements, transmission constraints, or competing generation from hydro or other sources.
The forward-looking target — what we actually train the model to predict — is the average absorption ratio over each forecast horizon window:
This horizon-aligned target construction matches how traders and analysts actually think about forward windows — the question is “what will the absorption regime look like over the next 5 days, not on Tuesday specifically.” Window-averaged targets are also more robust to single-day noise (a transmission outage, a data revision) than point predictions would be.
Each horizon has its own dedicated model, trained against its own target. Three models per ISO × fueltype combination — 42 models total across our 14 ISO × fueltype combinations.
We use XGBoost regression — gradient-boosted decision trees — for the absorption forecast. The choice is motivated by three properties of this problem.
Mixed feature types and non-linear interactions. Absorption regimes are driven by interactions between weather, demand, seasonality, and recent system state. Linear models cannot capture the regime-dependent relationships (e.g. the curtailment response to additional solar is steep up to a saturation point, then flat). Tree-based models handle these interactions natively.
Robustness to feature scale and distributions. Our feature set spans weather signals (continuous), seasonal indicators (cyclical), lagged absorption (bounded 0–1), capacity (monotonically increasing), and demand anomalies (z-scored). XGBoost does not require standardisation and is robust to outliers in any of these.
Honest treatment of missing data. Forecast-horizon features are null where forward weather forecasts haven't been published yet. XGBoost handles missingness explicitly during training rather than requiring imputation choices that introduce assumptions. Median imputation is applied as a fallback for production inference.
The model trains on 76 features per row, grouped into:
The feature set is deliberately rich on the demand side. The thesis of the model is that absorption regimes are driven by the interaction between supply (physics potential) and demand (regional gas-weighted degree days), not by either side in isolation. A sunny day with low cooling demand looks different from a sunny day with high cooling demand, and the model has the features to learn the difference.
For each ISO × fueltype × horizon combination, the model is trained on all historical days from June 2019 to the most recent date with a completed forward window (the strict near horizon target needs 5 forward days of actuals, mid needs 10, far needs 15).
We use walk-forward cross-validation for accuracy estimation: the model is retrained at each evaluation step using only data available before the prediction date, then evaluated on the next prediction. This is the only honest way to estimate forward-looking accuracy because it never lets the model see information it wouldn't have had in real-time deployment. We evaluate at weekly cadence (every 7th day) across the most recent 730 days of history, producing approximately 105 out-of-sample folds per model.
The walk-forward results are computed once per training run and persisted alongside the model. Reported accuracy metrics — MAE, R², seasonal breakdowns — come exclusively from these walk-forward folds, never from in-sample fit.
The XGBoost hyperparameters are tuned to balance fit against overfitting risk:
These choices are conservative. A more aggressive configuration would fit training data better and walk-forward worse. The published walk-forward accuracy is what subscribers actually experience.
Forecast accuracy is reported with seasonal breakdowns. The primary quality signal we track is spring MAE — mean absolute error during March, April, and May. This is the most important window for absorption forecasting because it covers the curtailment regime transitions: solar curtailment in CAISO ramps materially from March to May as physics potential grows faster than demand; wind curtailment in ERCOT and SPP shifts as the seasonal demand pattern rotates. A model that performs well in stable summer or winter regimes but fails during transitions has limited practical value.
Spring MAE above 0.15 triggers an internal review of the model’s demand features for the affected ISO × fueltype. This threshold reflects what we consider the limit of usable forecast quality for displaying live signals to subscribers.
The model outputs a forecast absorption ratio per ISO × fueltype × horizon. Three subsequent transformations produce the user-facing gas displacement number.
The absorption forecast is combined with the forward physics potential to produce expected dispatched generation:
This is the model's view of how much wind or solar electricity will actually reach the grid during each forward window. Crucially, this carries the curtailment regime forward — it is not a maximum-potential number, it is an absorption-adjusted forecast of real dispatched generation.
The TWh value is converted to gas-equivalent volume using the heat rate of marginal gas-fired generation. We use:
Combined:
where 7.28 Bcf per TWh is the dimensional conversion derived from the heat rate and HHV. Worked through: 1 TWh = 10⁶ MWh, requiring 7.5 × 10⁶ MMBtu of gas, which at 1.03 × 10⁶ MMBtu per Bcf equals 7.28 Bcf. This is what would have been burned to produce the same electricity from gas-fired generation.
For each daily model run, three predictions are produced per ISO × fueltype:
Each prediction carries its absorption forecast, derived estimated TWh, and derived Bcf displaced. All three are written to the predictions table with the model run timestamp and feature importance attached.
Day-over-day revisions are computed for every forecast metric — the change in today's forecast vs yesterday's forecast for the same horizon. These revisions are often more actionable than the absolute level itself: in commodity markets, the rate of change of expectations is what's being repriced. A 0.34 Bcf upward revision overnight on the near-horizon gas displacement forecast is a signal that something material has shifted in the inputs — weather forecast, demand outlook, or recent grid state — and that the market may not have fully repriced.
Every accuracy claim on the NatGas Edge platform comes from genuine out-of-sample predictions, not in-sample fit and not backtests.
On first deployment, we ran a complete walk-forward backfill: for each historical day in the validation window, a fresh model was trained using only data available before that day, then used to produce the prediction for that day. This produces a true historical track record — every prediction was made by a model that had no information about the outcome it was predicting.
From deployment forward, each daily model run produces predictions that are written to the predictions table before any future actuals exist. As actuals land over the following days and weeks, they are joined to the predictions in the downstream mart, producing live accuracy metrics computed only on genuinely forward predictions.
Every prediction record carries the feature importance of the model that produced it, persisted as JSON. This means the audit trail isn’t just “the model said X” but “the model said X because of these features, with these weights.” Over time this lets us track how the model’s reliance on different features evolves — whether autoregressive features are becoming more dominant, whether weather features earn weight during regime transitions, whether new fleet additions shift the importance distribution.
No feature in the training set incorporates future information about the target. Lagged features are strictly historical. Forecast-window features (weather, demand) are forward-looking forecasts, which is what the model would have had at prediction time. Actuals from the target window are never visible to the model during training or inference.
We document the model’s limitations openly because credibility depends on it.
The current supply-side weather signal is a proxy. A v2 enhancement will add fleet-weighted weather sourced from EIA-860 plant coordinates, which will sharpen physics potential calculations. Until then, the calibration model partly compensates by learning the empirical relationship between proxy weather and observed generation, but the limitation is real.
The absorption ratio captures total curtailment regardless of cause — demand-driven, transmission-driven, minimum-generation-driven, or competing-generation-driven. The model forecasts the aggregate accurately but does not attempt to attribute curtailment to specific mechanisms. Attribution would require transmission flow data, gas plant dispatch data, and ancillary service procurement data — pipelines we may add in future versions.
Gas displacement conversion uses the EIA-published US average of 7.5 MMBtu/MWh. ISO-specific heat rates would refine the calculation but the variation is typically within 10% and does not affect directional accuracy.
Renewable absorption regimes can shift with grid policy changes (transmission additions, new battery capacity, retirement of old gas units, regulatory changes to minimum-generation requirements) in ways the model only learns about after they occur. We retrain regularly and monitor accuracy continuously, but structural breaks in the underlying system cannot be perfectly anticipated.
Recent absorption levels are the strongest single predictor of forward absorption — this reflects the genuine persistence of curtailment regimes, not a modelling shortcoming. Weather and demand features earn their weight specifically at regime transitions, which is why spring MAE is the primary quality signal. The forecast is honest about what it is: an autoregressive prediction with weather/demand forcing, not a first-principles dispatch simulation.
The renewables signal is one input into the NatGas Edge composite weekly verdict, but it is the most differentiated. Public renewables forecasts stop at generation potential. Institutional fundamentals desks model absorption internally but their work is not accessible to retail. NatGas Edge produces forecast gas displacement in Bcf per ISO × fueltype × horizon — daily, with full revision tracking, audited live track record, and with the full methodology documented publicly.
This is the kind of signal that historically required a dedicated power-and-gas analyst working with several years of EIA data and ISO operational feeds. The reason it's not in retail tools is not that the analysis is hidden or proprietary — it's that the data engineering required to assemble and maintain the pipeline behind it is several months of work that no retail platform has historically been willing to invest in. We have done that work. This methodology is the audit trail.