Skip to content

Migrate to the PowerAnalytics 1.0 metrics API - #145

Open
PabloBotin wants to merge 9 commits into
Sienna-Platform:mainfrom
PabloBotin:feature/migrate-to-poweranalytics-new-api
Open

Migrate to the PowerAnalytics 1.0 metrics API#145
PabloBotin wants to merge 9 commits into
Sienna-Platform:mainfrom
PabloBotin:feature/migrate-to-poweranalytics-new-api

Conversation

@PabloBotin

@PabloBotin PabloBotin commented Jul 27, 2026

Copy link
Copy Markdown

Closes #144.

PowerAnalytics deprecated its pre-1.0 accessors (get_generation_data, get_load_data,
get_service_data, categorize_data, PowerData, …) in favor of the 1.0
Metric/ComponentSelector API — see the
"Old PowerAnalytics" notice
and tracking issue
PowerAnalytics.jl#28.
This moves PowerGraphics' internals across without breaking any public signature.

Why this is bigger than #144

Proving the migration correct meant comparing old and new API output per category, per
timestep — which is how the bugs below surfaced. That comparison has to hold on both
backends, and asserting it revealed the two ext/ recipes had drifted: each derived its own
defaults for fill, line width, draw order, title handling, empty input, save paths and
palette. Eight behaviors resolved twice, independently — the same drift that let the
bar-stacking fix in #140 land in one recipe and silently not the other.

Once those are resolved once in src/call_plots.jl and the recipes only draw, the backend is
purely a value, which src/backends.jl already modeled. The ten _plotly-suffixed functions
were then duplicating the public API without buying any dispatch, so they became a backend
key word.

This does not split into two PRs cleanly — the demand sign fix, the start_time/len fix and
the fuel-categorization rewrite live in the same functions as the backend unification, and the
migration's regression tests are written through the backend-parity harness. The commits are
ordered so the migration can still be read on its own: the first seven are the migration, the
last two the backend key word and review fixes.

1. Migration to the metrics API

plot_demand, plot_fuel and plot_results are all on the new API. plot_results no longer
constructs a PA.PowerData, and plot_powerdata(::PowerData) is deprecated to a shim.

One new export, get_demand_data, returning the demand data plot_demand draws with its
DateTime axis. Reading a single load metric does not give the same answer (see the sign bug
below) and there was no public route to those numbers — the report template needed one, and
every other public function returns a plot object.

2. backend is now a key word

plot_fuel(res)                                   # CairoMakie (default)
plot_fuel(res; backend = PlotlyLightBackend())   # PlotlyLight

The _plotly names still work but warn and forward. Passing both a _plotly name and a
backend key word raises an ArgumentError rather than letting one silently win.

Behavior users will notice:

  • Both backends now select from the whole load_palette palette, so more series get a
    distinct color before the cycle repeats. PlotlyLight's default colors change.
  • The default PlotlyLight save format is now html. A shared hardcoded "png" made every
    default-path save trip the extension-rewrite warning. An explicit format still wins.
  • CairoMakie non-stacked draw order now matches PlotlyLight.
  • WeaveExt = ["PlotlyLight", "Weave"] → "Weave". Neither the extension nor the report
    template touches PlotlyLight, so the old trigger withheld report from CairoMakie-only
    users.

docs/src/explanation/backend_parity.md records what the backends guarantee to render
identically and where they deliberately differ; test/test_backend_parity.jl enforces it.

Bugs fixed

  • Demand had the wrong sign and magnitude under controllable load formulations.
    calc_load_forecast applies an unconditional -1, but PowerSimulations stores the load
    parameter with a formulation-dependent sign, so a mixed system partially cancelled —
    measured −3604.307 against a true demand of +8853.873. _demand_data(::IS.Results) now
    resolves (calc_active_power, calc_load_forecast) per concrete load type.
  • The report template's Load table read calc_system_load_forecast directly, reporting a
    different number than plot_demand for the same results — the bug above, shipped in the
    template. It now goes through get_demand_data.
  • The plot_fuel net-load line now includes storage charging / source input, as its comment
    always claimed; the offset was computed and dropped.
  • plot_results(...; combine_categories = false) crashed with a MethodError. The docstrings
    also claimed false was the default when the code defaults to true — the documented
    default was the one that crashed.
  • plot_demand(sys; start_time, len) ignored those key words on the System path while the
    docstring advertised them.
  • plot_demand(...; save = dir) wrote the figure twice under two different names.
  • save_plot(p, "out.HTML") silently rewrote the path on PlotlyLight, which compared against
    ".html" exactly where CairoMakie lowercases first.
  • Two fuel-enum typos in the test mapping yaml (AG_BYPRODUCT, WOOD_WASTE_SOLIDS) that the
    old parser silently never matched.

Behavior notes

  • Fuel categorization keeps first-match-wins semantics (most specific type, then prime mover,
    then fuel), so no component is double-counted. Specificity comes from
    PA.parse_fuel_category, not from parsing display names.
  • Time windowing is applied locally by row slicing, because PowerAnalytics.compute
    mishandles window key words on simulation results.
  • ext_category entries in custom mapping yamls are ignored by the new selector parser, with
    a @warn.
  • The report template's Services table is gone — there is no new-API service metric.
  • Unmapped generators are reported in one aggregate @error instead of one per component.

Intentionally still on the old API

plot_demand(::PSY.System) (get_load_data(::System) has no new-API equivalent),
no_datetime on user-supplied DataFrames, and the deprecated plot_powerdata(::PowerData)
shims. PowerAnalytics still maintains the old API, so these keep working until a future
breaking release.

Testing

391 pass / 0 fail, with roughly 1,600 added lines under test/.

The migration is pinned by a numeric equivalence test against the still-exported old API,
covering every category across both UC and ED problems — storage In/Out, Curtailment,
Unserved Energy, Over Generation included — agreeing bit-identically
(maxabsdiff = 0.0), not merely within tolerance.

New test files: test_backend_parity.jl (the parity contract, both backends compared
directly), test_demand_semantics.jl (the sign bug and per-formulation metric resolution),
test_fuel_categories.jl (rule specificity and enum validation),
test_fuel_stack_behavior.jl (stack composition and the net-load line), and
plot_introspection.jl (backend-agnostic helpers so parity assertions are written once).

The public API was also exercised end-to-end outside the harness against real UC and ED
results — backend selection, the deprecated names, save formats and paths, palette assignment
and demand sign — with figures rendered on both backends for inspection.

Upstream issues

Workarounds are marked # TODO upstream and filed against PowerAnalytics: broken compute
time-window key words, stale get_subselectors export, inconsistent missing-result error
types, missing calc_system_slack_down/forecast metrics, and parse_generator_categories
returning nothing. The calc_load_forecast sign bug is still to be filed.

Pin the current fuel/demand data contract ahead of the PowerAnalytics
metrics-API migration: category naming (In/Out split, Curtailment, slack
display names), charging sign conventions, palette-first column ordering,
demand column naming, time-window and filter_func kwargs, and per-backend
series counts.
PowerAnalytics imports get_system from PowerSimulations, so the unexported
PA.PSI alias is unnecessary. Also extend the missing-system error to mention
loading results with populate_system = true.
The IS.Results path now computes Metrics.calc_load_forecast over the all_loads
selector (grouped into a single column renamed to "Load" so palette and label
behavior are unchanged); a user filter_func folds into the selector. Time
windows (initial_time/horizon, also spelled start_time/len) are applied by
local row slicing because compute rejects unknown kwargs and mishandles len on
simulation results in PA 1.4. The PSY.System path stays on the old
get_load_data API, which has no new-API equivalent. The dead isnothing guard
on the aggregated demand frame is replaced by an isempty check that can
actually fire.
Assemble the fuel stack from PowerAnalytics Metric/ComponentSelector
primitives instead of get_generation_data/make_fuel_dictionary/
categorize_data/combine_categories, preserving the exact column set, order,
names, and signs. Components are assigned to a single category by replaying
the old first-match-wins priority over the per-rule subselectors (the
independent new selectors would otherwise double-count, e.g. NG-CC vs
NG-Steam); generators fall back variable -> forecast parameter -> PowerOutput
aux; storage/sources split into '<category> In'/'<category> Out' with charging
flipped negative; slacks keep their BALANCE_SLACKVARS display names; unmatched
components go to 'Other' with an error log.

Also fix the net-load overlay to actually include storage charging by passing
the charging total as extra_load, update the test mapping yaml for the new
parser's strict fuel enums, and pin both behaviors with new tests.
…ta(::PowerData)

plot_results now owns its dict-of-DataFrames path (DateTime stripped per entry,
time axis from the first entry) instead of constructing PowerAnalytics.PowerData.
The plot_powerdata methods move to src/deprecated.jl as forwarding shims that
warn about removal in a future breaking release. combine_categories = false no
longer crashes: it plots one trace per stored column, and the docstrings now
state the actual default (true).
The Weave report template's tables now use the PowerAnalytics metrics API
(calc_active_power per fuel category, calc_system_load_forecast) instead of the
deprecated get_generation_data/get_load_data accessors; the Services table is
dropped since get_service_data has no metrics-API equivalent. Docstrings drop
the never-functional plot_fuel 'variables' kwarg, document the storage/sources
kwargs, and reference plot_results instead of the deprecated plot_powerdata.
The public API reference gains a hand-written Deprecated section.
@PabloBotin
PabloBotin marked this pull request as draft July 28, 2026 19:53
- plot_demand no longer crashes when a load type has no results; missing
  results skip to the "No load data found" path (now an ArgumentError)
- warn on the unsupported `variables` kwarg of plot_fuel instead of
  silently ignoring it
- warn when a custom generator mapping yaml contains ext_category keys,
  which the PowerAnalytics 1.0 selector parser cannot honor
- _combine_result_categories: unknown `names` entries raise an actionable
  ArgumentError; Vector{Symbol} accepted for the deprecated powerdata path
- docstrings: aggregate scope (System path only), time-window kwargs and
  aliases, aggregate-function return-shape contract
- _FuelRule stores type_name::Symbol to avoid per-supertype allocations
- test: old-vs-new numeric equivalence of fuel category traces
…names

Every plot function now takes `backend::PlottingBackend`, defaulting to
`CairoMakieBackend()`:

    plot_fuel(res)                                 # CairoMakie
    plot_fuel(res; backend = PlotlyLightBackend()) # PlotlyLight

The backend was already modeled as a value in src/backends.jl, so encoding it in
the function name doubled the public API without buying any dispatch. The ten
`_plotly`-suffixed functions keep working but warn and forward. Passing both a
`_plotly` name and a `backend` key word raises an ArgumentError rather than
letting one silently win, since the two would disagree about the renderer.

Eight per-plot behaviors that were resolved twice, once in each recipe, are now
resolved once in call_plots.jl and handed to the recipes through _PlotOptions:
fill default, line width, line style, draw order, title sentinel, empty input,
save path, and palette selection.

User-visible changes:

- Both backends select from the whole palette returned by `load_palette`, so
  more series get a distinct color before the cycle repeats. PlotlyLight
  previously drew from a narrower set, so its default colors change.
- `_default_save_format` dispatches on the backend, making the PlotlyLight
  default `html`. A shared hardcoded "png" tripped the extension-rewrite warning
  on every default-path PlotlyLight save. An explicit `format` still wins.
- CairoMakie non-stacked draw order now matches PlotlyLight.
- WeaveExt = ["PlotlyLight", "Weave"] -> "Weave". Neither the extension nor
  generic_report_template.jmd touches PlotlyLight, so the old trigger withheld
  `report` from a CairoMakie-only user.

The backend stubs dispatch per concrete backend. With `backend` defaulting to
CairoMakie, a PlotlyLight-only user reached a stub telling them to run `using
PlotlyLight` when they already had; each stub now names its own package, and the
CairoMakie one names the key word that selects the other backend.

Two save-path defects go with it. `_resolve_save_file` is now the single place a
save path is decided, and it replaces spaces in the title with underscores as
every entry point on main already did; centralizing the path had dropped that
for `plot_dataframe` alone. `_plot_demand!` read `:save` without removing it
from the key words it forwarded, so one call saved the figure twice under two
different names; it now strips `:save`, `:title`, and `:set_display` like the
other wrappers.

Tests: plot_introspection.jl reads rendered marks back out of both libraries so
value assertions run against either backend; test_backend_parity.jl enforces the
parity contract; test_demand_semantics.jl and test_fuel_categories.jl pin the
demand sign and the fuel-rule specificity. Suite is at 368 pass / 0 fail.

Docs gain explanation/backend_parity.md and a Change Backends how-to; the
orphaned explanation/stub.md is removed. Personal notes are ignored through the
user-level git ignore rather than this repository's shared .gitignore.
@PabloBotin
PabloBotin force-pushed the feature/migrate-to-poweranalytics-new-api branch from d9d0097 to b89dc10 Compare July 29, 2026 22:32
Route the report template's Load table through the new public
`get_demand_data` rather than `calc_system_load_forecast`, which reported the
requested instead of the served demand and disagreed with `plot_demand` under
controllable load formulations.

Delegate `_combine_result_categories` to `PowerAnalytics.combine_categories`
instead of reimplementing it, keeping only the actionable error on an unknown
`names` entry.

Move `seriescolor`, `column_labels`, `interval`, the scaled data matrix and the
net-sign classification into `_PlotOptions`, so neither recipe derives them
independently and the third spelling of the sign test disappears.

Lowercase the extension in the PlotlyLight writer so `.HTML` is recognized
rather than silently rewritten to a different path, and pin it with a test.

Hoist the shared "Accepted Key Words" documentation into
`_COMMON_PLOT_KWARGS` and interpolate it, replacing eight verbatim copies.

Keep `_report_plot_fuel` as a forwarding shim: report templates copied from an
earlier release call it positionally, so removing it would throw
`UndefVarError` on their next `report`.
@PabloBotin
PabloBotin force-pushed the feature/migrate-to-poweranalytics-new-api branch from b89dc10 to 39be6fe Compare July 29, 2026 22:34
@PabloBotin
PabloBotin marked this pull request as ready for review July 29, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update to work with v1.4.1 of PowerAnalytics.jl

1 participant