parameter sensitivity analysis at the cal/val sites, with latest version of spinet

[divine7022]

Report status and follow-up — 20 October 2026

Historical six-site SA report. This is distinct from the later 100-point local SA and 10-point global SA. The original report and review below are retained; its comparison/plot corrections are not all verified in the maintained report.

Implementation: SA coverage and orchard/prior checks #290, calibration specification #245, multisite fit #247, Russell integration #276, and practice comparisons #288. Outstanding checks include continuous US-Twt evaluation, equal-site summaries, observation-window alignment and the requested zero/missing plot conventions.

---

Parameter sensitivity of SIPNET at the six cal/val sites

Purpose

Rank SIPNET parameters by their contribution to output uncertainty at the cal/val sites, under the
priors we intend to calibrate from, and use that ranking together with the available observations to
set calibration priorities.

Partial variance measures contribution to output uncertainty under the specified priors. It does not
by itself establish that an observation can provide an independent constraint on a parameter. Those
two are kept separate throughout.

Design

Six sites, resolved to 25 site by treatment combinations so treatment contrasts are preserved:

site	system	treatments	period
Salinas	organic vegetable rotation	8 SOCS systems	2005-2011
Russell Ranch	irrigated and rainfed rotations	12 rotations	1992-2014
US-Twt	rice, Sacramento Delta	1, run over two windows	2010-2016, 2017-2023
US-Bi1	alfalfa, Delta	1	2017-2023
US-Bi2	corn, Delta	1	2017-2023
Modesto	almond orchard	1	2018-2019

Ten outputs at every run: soil carbon, above-ground wood, NEE, NPP, heterotrophic respiration, CH4,
N2O, N fixation, latent heat, soil moisture fraction.

Process representation is constant across runs: nitrogen cycle, anaerobic decomposition, flooding
and litter pool enabled everywhere, growing-degree-day phenology off. Whether a flux appears is left
to the simulated state. Two parameter choices also shape that outcome: the drainage fraction is held
at 20 per day at the 23 non-rice runs, which drains excess water within a timestep, and the methane
rate prior differs by about four orders of magnitude between the wetland and mineral soil
parameterizations. Methane appearing only at the rice runs reflects both.

31 parameters are varied per run, 32 at the two rice windows, spanning plant
physiology, stoichiometry, soil decomposition, the nitrogen cycle and soil moisture. Priors are taken
from the plant trait lookup where the crop is represented, the crop parameter draws otherwise, and
BETYdb or documented expert elicitation for the remainder. Sources are recorded per parameter in the
shipped prior table.

Method

One-at-a-time sampling. Each parameter is evaluated at five quantiles of its prior with the others at
their medians, and a response curve fitted through those five runs. Three quantities per parameter,
site and output:

• Prior spread, the interquartile range relative to the median. A property of the prior.
• Elasticity, the proportional change in output for a proportional change in parameter, at the
  median. Local model response, independent of prior width.
• Partial variance, the share of summed univariate output variance attributable to a parameter.

Limits on interpretation: interactions are not captured, parameters acting through the same model
term are not separated, and the partial variance denominator is the sum of univariate contributions
rather than total predictive variance.

Reading the figures

One figure per output. Parameters share a vertical axis in SIPNET documentation order; sites and
treatments run horizontally. Four panels: prior interquartile range relative to median; prior
coefficient of variation; partial variance on a fixed 0 to 1 scale; elasticity on a fixed diverging
scale, blue positive and red negative. Panels 1 and 2 span PFTs, so a bar shows a range where a prior
differs between them. Scales are shared across figures.

Grey cells mean the simulated flux is zero or negligible at that site, or the parameter is held
constant in that run. Grey describes the simulation, not the site. Cells beyond the elasticity scale
are marked.

Figure 1. sa_matrix_TotSoilCarb.png

Soil carbon. The two decomposition parameters carry the partial variance band and are negative in
elasticity. SLA, the largest plant contributor at 0.07, is near the bottom of the axis and
positive. The grey row is the drainage fraction, constant except at the two rice windows.

Results

Partial variance by output

Mean across runs.

output	first	second	third	runs
Soil carbon	soil_respiration_Q10 0.50	som_respiration_rate 0.27	SLA 0.07	25
NEE	soil_respiration_Q10 0.38	som_respiration_rate 0.25	SLA 0.15	25
N2O	soil_respiration_Q10 0.38	n_volatilization_rate 0.38	n_leaching_frac 0.07	25
CH4	soil_methane_rate 0.49	soil_respiration_Q10 0.43	water_drain_frac 0.08	2
NPP	SLA 0.40	psnTOpt 0.23	half_saturation_PAR 0.13	24

The CH4 row averages the two runs where methane is produced and is not comparable across rows.

Grouping decomposition, nitrogen and moisture parameters as soil, and photosynthesis, phenology,
stoichiometry and tissue parameters as plant: soil parameters account for 86 percent of soil carbon
variance, 90 percent of N2O and 69 percent of NEE; plant parameters account for 99 percent of NPP.
The two decomposition parameters account for 77 percent of soil carbon variance and the plant side
for 14 percent.

Figure 2. sa_matrix_CH4_flux.png

Methane. Twenty-three of 25 runs are grey; the simulated flux is zero or negligible there. At the
two rice windows the methane rate and the soil respiration temperature sensitivity are the largest
contributors. The drainage fraction contributes at the 2010-2016 window and is negligible at the
later one.

Figure 3. sa_matrix_N2O_flux.png

Nitrous oxide. All 25 runs carry a decomposition; nothing is grey except the constant drainage
row. Variance sits on the volatilization rate and the soil respiration temperature sensitivity.
Volatilization is positive, leaching and photosynthetic capacity negative. 93 of 777 cells exceed
the elasticity scale.

Figure 4. sa_matrix_NPP.png

NPP variance is dominated by plant parameters, led by SLA and psnTOpt.

Effect of the calibration target

Soil carbon can be fitted against the stock, its change over the record, or the treatment minus
control difference. The same seven amended Salinas systems, decomposed three ways:

parameter	stock	change	treatment minus control
soil_respiration_Q10	0.34 (1)	0.25 (1)	0.11 (3)
SLA	0.18 (2)	0.24 (2)	0.35 (1)
som_respiration_rate	0.17 (3)	0.17 (3)	0.02 (10)
half_saturation_PAR	0.06 (4)	0.08 (4)	0.12 (2)
fracLitterRespired	0.03 (8)	0.03 (9)	0.08 (5)

Plant parameters account for 36 percent of variance in the stock, 49 percent in its change and 72
percent in the contrast.

Two systems at the same site share climate, soil and soil parameter values, so differencing cancels
much of the soil signal and retains what differs between systems, which is the amendment and the
cover cropping. som_respiration_rate falls from 0.17 to 0.02.

Figure 5. sa_matrix_deltaSOC.png

Change in soil carbon over each record. Decomposition parameters still lead; the plant band is
stronger than in the stock figure.

Figure 6. sa_matrix_deltaOfDeltas.png

Salinas treatment minus control, seven amended systems against the unamended control. SLA is the
largest contributor; the base decomposition rate has largely cancelled. The litter respired
fraction rises from eighth on the stock to fifth.

The contrast isolates the management effect but is dominated by plant parameters, and the current
observation set contains no yield or biomass measurement bearing on them.

Simulated soil carbon trajectories

Median-run change over each record.

site	change, kg C m-2	fraction of simulated stock
Salinas, unamended control	-0.54	-8%
Salinas, seven amended systems	+1.05 to +1.62	+13 to +20%
Russell Ranch, all 12 rotations	-2.25 to -4.47	-26 to -58%
US-Bi1, US-Bi2	-1.62, -1.13	-8%, -5%
US-Twt	-3.90, -2.10	-20%, -16%
Modesto	-0.12	-2%

Every unamended run loses carbon; only the amended Salinas systems gain.

Model against observations

target	model	observed	discrepancy
Salinas SOC, unamended control, yr 2-8	-5.4 Mg C ha-1	-7.8 Mg C ha-1	same sign, same order
Salinas SOC, treatment minus control	+15.9 to +21.6 Mg C ha-1	-1.8 to +5.5 Mg C ha-1	3.7x to 27x, sign reversed at 2 of 7
Russell Ranch SOC, 11 rotations	-22.5 to -44.7 Mg C ha-1	-5.6 to +9.7, mean +0.9	model declines, observations approximately stable
US-Twt CH4, 2010-2016	144 g C m-2 yr-1	8.0 to 17.9, mean 12.4	about 12x
US-Bi1 and US-Bi2 CH4	0.0001 and 0.0 g C m-2 yr-1	0.49 to 2.58 g C m-2 yr-1	model near zero
Modesto N2O	1.8 g N m-2 yr-1	54 chamber measurements, about 0.06 annualized	about 28x

Russell Ranch is the largest absolute discrepancy among the soil carbon comparisons. The model loses
22 to 45 Mg C ha-1 over 1993 to 2012 while the observations are approximately stable. This limits
the use of these trajectories for quantitative calibration until the discrepancy is understood.

At Salinas the simulated unamended decline is within about 30 percent of the observed one, while
every amended system moves from an observed loss to a simulated gain. Observed soil carbon declines
in all eight systems over this period, including the composted ones; only the between-system
difference is positive, and it is small. The model retains 16 to 22 Mg C ha-1 of the 37 Mg C ha-1 of
organic carbon applied as compost. About a quarter of the upper bound is cover crop rather than
compost: only sys2 shares the control's planting schedule, and it retains 15.9.

The agreement on the unamended decline should not be read as support for the carbon balance.
Simulated plant carbon input at Salinas is 1.5 to 8 Mg C ha-1 against 33 to 65 observed, and
simulated NPP has a median of 55 g C m-2 yr-1 across the 25 runs, low for irrigated annual cropping.
This also conditions the ranking above: plant parameters contribute little to soil carbon variance
in a model state where simulated production is itself small.

At US-Twt the field is simulated as ponded for most of the year at every sampled drainage value; the
beta(2, 30) prior places all five knots between 0.008 and 0.170 per day. The staged irrigation record
includes 23 of 137 events between November and March, outside the rice season. Simulated methane
differs between the two US-Twt windows, 491 against 144 g C m-2 yr-1, but the windows also differ in
soil carbon by a factor of 1.5 and in total irrigation by a factor of 5, on different initial
conditions, and methane is linear in soil carbon in the model, so the difference cannot be attributed
to off-season water.

At the two Bouldin Island sites simulated methane is near zero against an observed flux. Those runs
carry a mineral rather than a wetland soil parameterization.

Calibration priorities

Target. Fit against soil carbon stock or its change. Soil parameters account for 86 percent of
the variance in the stock and 81 percent in its change across the 25 runs; at the seven amended
Salinas systems, 64 and 51 percent. Use the treatment contrast as a diagnostic rather than the
fitting target: plant parameters account for 72 percent of its variance and the current observation
set does not constrain them.

Free.

parameter	basis
soil_respiration_Q10	largest contributor to soil carbon, NEE and N2O variance
som_respiration_rate	second largest on soil carbon and NEE; the only soil parameter here with a fitted rather than carried-over prior

These act through the same respiration term and cannot be separated well from soil carbon
observations alone. Expect a correlated pair rather than two independent estimates. If a single rate
is preferred, fix the temperature sensitivity at its prior median and let the base rate absorb the
adjustment.

Fix.

parameter	basis
SLA, psnTOpt, Amax, half_saturation_PAR	high partial variance on NPP and on the treatment contrast, but the current observation set contains no yield or biomass measurement providing an independent constraint. Freeing them against soil carbon alone allows a production bias to be absorbed into a production parameter.
fracLitterRespired, turn_over_time	fifth and sixth on the contrast; priors not independently sourced. Candidates once the amendment response is fitted directly.
remaining parameters	no observation in the current set bears on them

Trace gases. Both are single-site targets and should not be folded into a joint fit with soil
carbon.

n_volatilization_rate accounts for 38 percent of N2O variance, and Modesto is the only site with
N2O observations. N2O can be treated as a separate calibration target, subject to two limits: the
records reduce to two calibration cells, and the model-observation discrepancy is large enough that
fitting may require substantial movement from the prior.

soil_methane_rate should not yet be fitted. Under the current water balance the field is ponded
for most of the year at every sampled drainage value, so fitting the methane rate risks compensating
for the water-balance error rather than identifying methane production. At the Bouldin sites the
soil parameterization would need to change first.

Conditioning on the fitted rates. The Salinas compost retention discrepancy is larger than the
free parameters can absorb within their priors, so fitted decomposition rates will partly compensate
for a carbon input or retention error. Report them with that conditioning stated and revisit once the
amendment representation is settled.

Limitations

One-at-a-time sampling does not resolve interactions, which bears most on the soil_respiration_Q10
and som_respiration_rate pair. Thirty-one of about 77 model parameters are varied. Two of the
remainder bear on conclusions here: soil water holding capacity is 12 cm at every site including the
peat rice paddy, and governs when the model ponds; and the drainage fraction is constant at the
non-flooded sites and sampled only at the rice site. Fine root respiration was held at its default.

Observed soil carbon carries uncertainty the comparison table does not show. At Salinas the standard
error on a treatment contrast is 4 to 6 Mg C ha-1 across four replicates, and six of seven observed
contrasts are within one standard error of zero. The simulated contrast of 16 to 22 is outside that
range, but the observed values should not be read as precise bounds. The model integrates a single
soil pool not depth matched to the 0-30 cm observations.

Initial soil carbon at the three Delta sites is from a gridded product rather than site measurement,
and at US-Twt is roughly half the published peat stock, so the simulated declines there are not yet
interpretable. Delta initial conditions and the rice water balance are open items tracked separately.

Outputs

Under /projectnb/dietzelab/ccmmf/usr/akash/calval_sa_v3_20260821; the README records model version, configuration, run list and limitations.

• analysis/sa_table.csv -- one row per run, output, PFT and parameter: prior coefficient of
  variation, elasticity, partial variance, and the simulated magnitude of that output.
• analysis/delta_soc_table.csv, analysis/delta_of_deltas_table.csv -- the same decomposition on
  the change in soil carbon and on the treatment minus control change.
• analysis/figures/ -- twelve figures, one per output plus the two delta targets.
• pfts/prior_table.csv -- priors with sources.
• runs/run_table.csv -- the 25 runs with windows, PFTs and input provenance.

[dlebauer]

Overall, this is a very helpful and clearly explained report and interpretation of the OAT sensitivity analysis. However, several calculations and assumptions need to be revisited - including mean responses, and alignment of model output with observations (in time, and quantity) - should be updated before focusing on changing priors.

As always, please create an issue to track any work or requested changes that are not immediately necessary. e.g. do: shifting to comparing outputs with broader syntheses, defer: detailed rationale of associated assumptions.

The currently separate US-Twt simulations and evaluations should be joined into a single time series. This change was requested upstream in ccmmf/cal-val-data#2 and #261.

Quantities

Prefer comparisons of changes in SOC and AGB to reduce conflation of model behavior with initial conditions.

For CH4, prioritize agreement in direction and order of magnitude, but do not focus too much on parameterization because we know that relevant inputs will change. Planned changes include: (1) site-specific soilWHC initial conditions, which are currently fixed, and (2) removal of irrigation events that are inconsistent with known agronomic practices, such as irrigating a fallow rice field. We cannot wait for all of these upstream changes before continuing, and they may not be sufficient, so treat the current comparisons and parameter conclusions as provisional, with stated assumptions and limitations.

For N2O, 1) only use chamber measurements to evaluate treatment comparisons and 2) Align model output and observations by measurement date and aggregate chamber replicates within treatment and date because the observations are chamber records from nine dates. The reason not to compare magnitudes is based on CARB guidance and well-documented limitations of the chamber method, (it is widely done but) N2O is too spatially heterogeneous to allow correct scaling of chamber measurements to field-level annual estimates. Also, do not interpret a sum across nine measurement dates as an annual flux.

Statistical considerations

A mean across sites is more informative than a mean across runs. When computing summary statistics, including the likelihood for calibration, use explicit site weights so that sites with more treatments do not have disproportionate influence. The same principle applies within treatment time series.

---

On separating information used in parameterization from information used in evaluation:

Even when observations are held out from calibration, the same studies may have influenced model structure, default parameter values, prior ranges, or syntheses used to set them. This is especially difficult for SOC because the limited long-term records are valuable for both model development and evaluation. See the discussion of "double dipping" in Gelman et al., Bayesian Workflow, although strictly preventing indirect information reuse is often impractical in this domain.

If the prior syntheses provide usable constraints, use them rather than comparisons with the validation sites to set plausible directions and ranges. These are being assembled by @AritraDey-Dev as described in #226 and #164. Note any known overlap with our sites; a formal correction is unnecessary unless an overlapping study has substantial influence. Use the site comparisons primarily to evaluate performance and diagnose discrepancies, and limit their use in setting priors or changing model structure.

Plots

I like the addition of the IQR plots. Please order the plots as IQR/CV, elasticity, and variance contribution. This order shows how prior spread and sensitivity contribute to output variation - relative output SD is approximately |elasticity| × CV.

Add parameter half-eye and sensitivity plots. I can prepare a first draft of these.

Show parameter groupings on the y-axis and treatments grouped within sites on the x-axis. Replace parameter names with human-readable labels; I think these are available in PEcAn standard_vars or the BETYdb variables table. Replace site IDs with site names, etc.

For the heat maps, especially variance contribution, adjust the color scale so that lower values are easier to distinguish. The distributions appear zero-inflated with a long right tail. Distributing colors across quantiles is one option. If quantile breaks are used, label them explicitly because equal color steps will not represent equal changes in variance contribution.

Use white for zero or negligible values and grey for NA, missing, inactive, or undefined values. This is particularly important for methane, where white for zero or negligible values would make the important parts of the plot easier to identify.