Repository navigation
Using group_initial_split() with small group will fail even if adjusting the prop parameter? #534
Description
The problem
Summary: group_initial_split() fails often with small-frequency groups even if adjusting prop to reflect the small-frequency group?
I'm using group_initial_split() with a small number (4) groups. As I have one group with low frequency (10%), my intuition was that by setting prop=0.9, this group would be selected within the training sample. However, I get very often (around 70%) error messages such as:
#> Error in
group_mc_cv():
#> ! Some assessment sets contained zero rows
#> ℹ Consider using a non-grouped resampling method
How come this happens even if I adjusted prop? This fails even if I get the exact proportion of the group (1-freq(small_group))!? Am I misunderstanding the prop argument?
Thanks!
Reproducible example
library(rsample)
dat <- data.frame(group = sample(LETTERS[1:4], prob = c(0.3, 0.3, 0.3, 0.1), replace = TRUE, size=1000),
x = rnorm(1000))
table(dat$group)
#>
#> A B C D
#> 340 270 298 92
set.seed(123)
dat_split <- group_initial_split(dat, group, prop=0.9)
#> Error in `group_mc_cv()`:
#> ! Some assessment sets contained zero rows
#> ℹ Consider using a non-grouped resampling method
# This will fail about 80% times:
set.seed(1234)
mean(sapply(1:100, \(x) inherits(try(group_initial_split(dat, group, prop=0.9), silent = TRUE), "try-error")))
#> [1] 0.79Created on 2024-09-08 with reprex v2.1.1
Activity
hfrick commentedon Sep 12, 2024
Hi @MatthieuStigler
From the docs:
while you are
Since groups as a whole get allotted to training or testing, they can't be all represented in the test set, otherwise there would be no observations left for the training set.
Stratification (as opposed to grouped resampling) aims to ensure that the proportion of each group is the same in the training and testing set as it is in the full dataset. So if you have a small group and want a training and testing set which both contain all groups, including that small group, stratification is typically what you want to use. This can be done with the
strataargument forinitial_split(), see example below.Does this help?
Created on 2024-09-12 with reprex v2.1.0
MatthieuStigler commentedon Sep 12, 2024
Hi @hfrick
thanks a lot for the answer. Sorry, that last statement was a bit misleading (I meant that by running K times, I want to each time one group in the test sample), so I removed that part.
The main question remains: how come, having one group with frequency 0.1, setting
prop=0.9fails consistently (instead of attributing the 10% group in the test sample).Thanks!
hfrick commentedon Sep 13, 2024
Ah, I see. Thanks for clarifying!
I would say this could be loosely answered with "the error happens because we are sampling, not optimizing". In your example, we have 4 groups with one group about the size of the test set. So a grouped split with
prob = 0.9only works if we assign that smallest group,D, to the test set. But we have 4 to choose from, so it should fail in 3/4 of the attempts.If you increase the number of attempts in your last illustration, you should be able to see it move towards 0.75.
[-]Using group_initial_split() with small group will fail even if adjusting the `prop` parameter?[/-][+]Using `group_initial_split()` with small group will fail even if adjusting the `prop` parameter?[/+]MatthieuStigler commentedon Dec 17, 2025
hello! I tried with the latest version of rsample, and can reproduce the problem?
Created on 2025-12-17 with reprex v2.1.1
EmilHvitfeldt commentedon Dec 17, 2025
Perfect sorry about that!
hfrick commentedon Jan 29, 2026
This is not a bug, this is how the sampling works. If you have follow-up questions to my explanation above, please ask :) I'm going the close the issue though.