CapeCod.predict infers the fitted grain from key_labels: should that inference exist?

[ppcvote]

Split out of #1265 at @henrydingliu's request. No bug here, this is the design question that issue kept running into.

There are two ways to say "fit by line of business, apply per company":

# A: the parameter
cl.CapeCod(groupby="LOB").fit(tri, sample_weight=prem)

# B: aggregate the training data and let predict() work out what happened
model = cl.CapeCod().fit(
    tri.groupby("LOB").sum(), sample_weight=prem.groupby("LOB").sum())
model.predict(tri, sample_weight=prem)

B works by diffing key_labels between sample_weight and apriori_ and regrouping if they differ (capecod.py:325). That inference is where the #1265 bug lived.

The two routes agree, once development is grouped consistently (clrd, comauto):

route	apriori
A, Development() ungrouped	0.5602627308
A, Development(groupby="LOB")	0.5689995797
B	0.5689995797

One correction to what I wrote in #1265: I presented A and B as giving different aprioris. The difference is entirely the development grain, not the CapeCod grain.

So: with groupby available and giving the identical answer, does predict() need to infer the grain at all? As I see it:

1. Leave it. It is the behaviour [BUG] CapeCod predict at different index grain #400 asked for and it works once CapeCod.predict discards the fitted apriori when the model was fit on exactly one fewer index level #1265 is fixed.
2. Keep it, document that A and B are equivalent, and warn when the inference fires.
3. Deprecate the inference in favour of groupby.

I lean 2. The inference is convenient, but it currently happens invisibly: a user who calls predict() on a 775-company triangle gets a 6-row apriori_ back and nothing says why. A warning turns a silent regrouping into a visible one.

I can implement whichever you prefer.

[henrydingliu]

thanks for raising! i'm leaning towards 2 as well. before we proceed, can you please investigate whether 2 is happening invisibly on other estimators? just the most common ones (Development, Chainladder, BornhuetterFerguson)

[ppcvote]

Checked all three. The inference is unique to CapeCod, and the reason looks structural rather than accidental.

Fitting at LOB grain on clrd and applying at company grain, on the #1275 branch:

estimator	fitted total	applied total	what comes back
Development.transform	n/a	n/a	ldf_ stays 6 rows, values identical to the fitted ones
Chainladder.predict	152,598,500.44	152,598,500.44	ultimate_ at the caller's 775 rows
BornhuetterFerguson.predict	154,218,735.84	154,218,735.84	ultimate_ at 775 rows, key_labels unchanged
CapeCod.predict	153,611,521.17	153,611,521.17	apriori_ at 6 rows, down from 775

None of the three re-estimates anything at predict time:

• Development.transform attaches the fitted ldf_ unchanged and lets it broadcast.
• Chainladder.predict is super().predict plus _get_ultimate, which consumes ldf_.
• BornhuetterFerguson.predict is just super().predict, and _get_benktander_aprioris reduces to sample_weight * self.apriori. The apriori is a parameter, never estimated, so there is no grain to infer. ExpectedLoss inherits the same method.

CapeCod is the only one of the four whose parameter is estimated from (X, sample_weight) at predict time, in _get_capecod_aprioris, so it is the only one that has to pick a grain. That is also why it is the only one whose returned object changes shape: apriori_ comes back at the fitted grain rather than the caller's, which is the invisible part.

Grepping key_labels across the package agrees: outside the broadcasting in core/dunders.py, capecod.py:325 is the only place where a key_labels difference drives a decision.

So option 2 is a warning on one code path, not a cross-cutting change. Happy to write it whenever #1275 lands.

[henrydingliu]

thanks for the investigation! sounds like capecod is doing something unique. should the warning in option 2 be a deprecation warning (i.e. we'll eventually implement option 3)?

and do you think the broadcasting in the other estimators is robust?

[ppcvote]

Both answered, and the second question turned something up.

Deprecation or plain warning. I would start with a plain warning rather than a DeprecationWarning, because the two routes are not drop-in equivalent today. On clrd, comauto:

route	apriori
CapeCod(groupby="LOB") with Development() ungrouped	0.5602627308
CapeCod(groupby="LOB") with Development(groupby="LOB")	0.5689995797
pre-aggregated training data	0.5689995797

Someone who reads a deprecation notice and swaps to groupby gets a different number unless they also group Development. A DeprecationWarning commits to removing #400's documented path before that migration note exists. A plain warning fixes the invisibility now and costs nothing if you later decide on option 3, at which point the same call site becomes the deprecation.

Robustness of the broadcasting. Loud in most directions, silent in two, and both silent ones are in Chainladder. Fitting on clrd and then predicting:

situation	Chainladder	BornhuetterFerguson	CapeCod
fit at LOB grain, predict per company, all groups seen	correct	correct	correct with #1275
prediction data carries a group the model never saw	returns 643 of the 775 rows passed in, no warning	ValueError	ValueError
prediction data is a strict subset of the fitted groups	correct	correct	correct
fit per company, predict at LOB grain	6 rows in, 775 rows out, no warning	ValueError	ValueError

The ValueErrors are all operands could not be broadcast together with shapes (775,1,10,1) (643,1,10,1): safe, but they say nothing about which index values caused it.

The two Chainladder cells are the real gap. In the first, the 132 rows it drops are exactly the held-out LOB, verified row by row, so anything joined to the result afterwards has a silent hole. In the second, the returned object carries the fitted model's key_labels (['GRNAME', 'LOB']) rather than the caller's (['LOB']), and the values are not the fitted ultimates either. Both trace to MethodBase.predict:

X_new = X_new + (self.X_.val_to_dev().iloc[0, 0].sum(2) * 0)

which aligns the caller's triangle onto the model's index instead of checking that the two agree.

Happy to file that separately so it does not get lost behind the warning discussion.

[henrydingliu]

Someone who reads a deprecation notice and swaps to groupby gets a different number unless they also group Development.

I miscommunicated what I meant by deprecation. I was actually thinking deprecating the "inference at predict" behavior, i.e. a priori is locked during fitting; predicting on a more granular triangle will simply follow existing bf broadcasting path. reading #400 again, i didn't see inference at predict as the stated goal, though it became the resolution.

but such a deprecation needs to be more broadly socialized. so i'm good with a plain warning.

Happy to file that separately so it does not get lost behind the warning discussion.

yes please

[ppcvote]

Correction to my last comment: the causal claim in it is wrong. I read predict() instead of stepping through it, and I named a line that does nothing relevant to either case.

I blamed X_new = X_new + (self.X_.val_to_dev().iloc[0,0].sum(2) * 0). Both cases are actually the next call down, self.intersection(X_new, X_new.ldf_) at base.py:111. On main at 8a47051:

case 1, fit on 5 LOBs, predict on all 6
  after val_to_dev                       (775, 1, 10, 10)  ['GRNAME', 'LOB']
  after the '+ (self.X_ ... * 0)' line   (775, 1, 10, 10)  ['GRNAME', 'LOB']   the line I blamed
  after intersection()                   (643, 1, 10, 10)  ['GRNAME', 'LOB']

case 2, fit per company, predict at LOB grain
  ldf_ after fit                         (775, 1, 1, 11)   ['GRNAME', 'LOB']
  after intersection(): the triangle     (6, 1, 10, 10)    ['LOB']
  after intersection(): its ldf_         (775, 1, 1, 11)   ['GRNAME', 'LOB']

So it is one line with two failure shapes, which is a tidier story than the one I told. In case 1 intersection() narrows both operands and the caller loses rows it passed in. In case 2 it narrows only one: the caller's six LOB values already cover every LOB in the 775-row ldf_, so nothing is dropped on that side, and predict() returns an object whose triangle is at the caller's grain and whose ldf_ is at the model's. Chainladder._get_ultimate then consumes that stale ldf_, which is where the 775-row ultimate_ I reported comes from.

The behaviour I described is unchanged and still reproduces on current main. Only the explanation was wrong.

[henrydingliu]

thanks for the correction! feel free to put in a PR for the warning.

[ppcvote]

Opened #1306 for the warning.

One design note that is easier to raise here than to discover in review: CapeCod(groupby=...) reaches the same branch, because with groupby="LOB" the fitted apriori_ sits at ['LOB'] while sample_weight is still at ['GRNAME', 'LOB']. Warning there would fire on someone who already stated the intention, so the warning is scoped to self.groupby is None. Happy to change that if you would rather it fire either way.

[henrydingliu]

thanks for finding this nuance!

i agree that it doesn't make sense to fire a warning at someone who stated the intention. but that led me to think about what that intention is. the current implementation of capecod re-estimates apriori_ on every inference. there is essentially no model persistence. as in, i can't save the capdecod from last quarter and reapply it this quarter.

basically, based on our intention to fire this warning around index levels, should we actually be firing off a warning for every predict. and if that's the case, what does it mean about the method overall?

[ppcvote]

Chased what "no persistence" actually amounts to, and it is sharper than the index-level case.

predict returns a third number, neither the fitted apriori nor a clean refit. On ukmotor, fitting on the prior diagonal and predicting on the current one, with identical index levels so the inference branch never fires:

	apriori
fitted on the prior diagonal	0.6572660657
returned by predict	0.6894525318
a plain refit on the new data	0.7062253126

It re-estimates from the new X and sample_weight while keeping the fitted ldf_, so what comes back is neither the model you saved nor the model you would build today.

On warning every time, I would say no. A warning that fires on every call is one people switch off, and this is a property of the method rather than a mistake at a call site. #1306 is about a specific surprise, that apriori_ comes back at a different grain from the one passed in. "Every predict re-estimates" reads to me as documentation, and possibly API, rather than a warning.

Which brings up the other half of your question. Persistence already exists as a manual pattern, and what it costs is the more interesting part:

stored = cc.detrended_apriori_
bf = cl.BornhuetterFerguson(apriori=1.0).fit(prior, sample_weight=exp_prior * stored)

That reproduces CapeCod exactly on the fitting data, 78871.9279 either way, which is itself the answer to what it means about the method: freezing the apriori for reuse is just running BornhuetterFerguson with CapeCod's fitted expectation baked into the exposure. Persistence and re-estimation are the same fork, and taking one means opting out of the thing CapeCod does. Carried to the next diagonal, over the six origins the stored apriori covers, it gives 80242.93 against CapeCod.predict's 80774.45, a 0.66% difference.

It breaks where you would most want it, though. The newest origin has no stored apriori, so exp_now * stored is NaN there, and the result is an ultimate of 6283.00 for 2013, exactly the reported to date, with no IBNR at all. ibnr_ is NaN for that origin but ultimate_ holds no NaN, so nothing on the object signals it.

So if persistence is worth making first class, the hard part is not storing the apriori. It is deciding what happens to origins the stored one never saw, which is presumably what trend and decay would have to answer.

[henrydingliu]

On warning every time, I would say no

agreed. my point is, are both ways of saying "fit by line of business, apply per company" the same, i.e. "fit by line of business, refit per company". and if that's the case, is the warning we are thinking of firing applicable to all paths? and since we don't want to fire warnings every time, do we just want to include it into the docstring

Persistence already exists as a manual pattern

this is going down a path that i think we eventually have to go down. in practice, the a priori for a BF method is often determined based on some historical chainladder result. for example

1. find paid and incurred chainladder ultimates
2. calculate average ultimate loss ratio from the last 5 years excluding latest
3. use the result from step 2 as a prior, calculate BF ultimates
4. weigh between cl and bf ultimates

this package currently doesn't support putting this entire analysis into a pipeline. something we'll have to revisit in the future

It breaks where you would most want it, though. The newest origin has no stored apriori

detrended_apriori_ is origin-specific because the trend and olf vectors are origin-specific. the origin-robust version of the manual bf path would be

stored = cc.apriori_
bf = cl.BornhuetterFerguson(apriori=stored).fit(prior.trend(some_trend), sample_weight=exp_prior * olf_vector)
bf_ultimate = bf.ultimate_.trend(1/(1+some_trend))

my proposal is this, we raise a new issue for broader community input around whether capecode should behave as is or behave like bf(apriori=capecod.apriori_ / sample_weight).

• if consensus is 'as is', then we make explicit disclaimer in the docstring around the re-estimation behavior
• if consensus is 'no re-estimation', then a bigger refactor effort would be needed.

let me know your thoughts

[henrydingliu]

@ppcvote bump

[ppcvote]

Sorry for the delay. Ran both questions rather than answering from memory. Everything below is on current main (6b51f649), after #1275.

Are the two routes the same? Yes, and more strongly than the issue body claims, which was one apriori scalar on comauto. On the full clrd triangle, 775 companies, with development grouped consistently:

route	total ultimate
A, CapeCod(groupby="LOB")	153,611,521.1665
B, aggregate then predict	153,611,521.1665

Elementwise they agree to float64 precision: max absolute difference 1.86e-9 against values of order 1e5 to 1e7, with NaN in the same 1540 cells. Both return apriori_ at ['LOB'], six rows. Leave development ungrouped and A comes back 152,668,727.5764 with an elementwise gap up to 180,547.70, so the divergence is the development grain, not CapeCod.

Is the warning applicable to all paths? I started out thinking route A had nothing to warn about. That is wrong. A's fit() never reaches the branch, it sets expectation_ directly at capecod.py:240. But persist a groupby-fitted model and predict with it, which is the case your question is actually about, and A hits the identical condition:

A = cl.CapeCod(groupby="LOB").fit(d, sample_weight=prem)
set(prem.key_labels) - set(A.apriori_.key_labels)   # {'GRNAME'}, so capecod.py:324 fires

So yes, for any predict, whichever route built the model. That also fixes what the warning is saying: not that the answer changed, both routes give 153,611,521.1665, but that apriori_ comes back coarser than the triangle passed in.

On re-estimation: agreed, docstring rather than a warning.

On your proposal: for it, with a caveat I want to put before the evidence rather than after. What follows is in-sample, the same X for the fit and for the reconstruction. It is not the persistence case you are describing, and I have not tested either form against a later triangle with a refreshed diagonal.

With that said, BornhuetterFerguson(apriori=1.0).fit(X, sample_weight=exposure * cc.detrended_apriori_) reproduces CapeCod exactly on comauto, othliab and wkcomp, elementwise difference 0.0. That is close to tautological, since fit() is expectation_ = sample_weight * detrended_apriori_, so it shows the BF syntax can express the formula, not that a stored apriori survives a period roll. Worth someone testing that before the refactor branch gets sized.

Your origin-robust form using apriori_ with trend and olf is the right correction to what I wrote.