From 380ea8f1e007be9cf3585352378ce3040291f0a9 Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Mon, 27 Jul 2026 16:44:28 +0100 Subject: [PATCH 01/10] start module skeleton --- workflow_piping/index.qmd | 42 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 workflow_piping/index.qmd diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd new file mode 100644 index 0000000..00eee8d --- /dev/null +++ b/workflow_piping/index.qmd @@ -0,0 +1,42 @@ +--- +title: "Workflow design and piping" +description: "" +author: "Etienne Bacher" +date: "2026-07-27" +categories: [r] +difficulty: Beginner +format: + html: default + revealjs: + output-file: index-slides.html +execute: + warning: false + message: false + freeze: auto +editor: + markdown: + wrap: 72 +--- + +# Introduction + +## Example where we don't use the pipe + +- assign each function output to an object + + - drawback: either create plenty of temp object names, or overwrite the same object again and again, in which case we have to be careful if we want to rename this object later. + +- alternative: use nested function calls + + - drawback: harder to create because we have to start from the most nested function and progressively go outwards + + +## Introducing the pipe + +[ add a quarto callout explaining the "history" of `%>%` and `|>` so that people know why `%>%` still appears in a lot of docs and examples ] + +- explain `|>`: how to use, constraints (R >= 4.1, RHS must be a function call with `()`) + +- explain `_`: how to use, constraints (must be associated to a named argument, can't appear more than once in the RHS) + +- rewrite the example above with the pipe and `_` \ No newline at end of file From 0c2560b6155847473496b16773f6f5be7199e0c2 Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Mon, 27 Jul 2026 16:45:46 +0100 Subject: [PATCH 02/10] typo --- workflow_piping/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index 00eee8d..5bb7fda 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -37,6 +37,6 @@ editor: - explain `|>`: how to use, constraints (R >= 4.1, RHS must be a function call with `()`) -- explain `_`: how to use, constraints (must be associated to a named argument, can't appear more than once in the RHS) +- explain `_`: how to use, constraints (R >= 4.2, must be used as a named argument, can't appear more than once in the RHS) - rewrite the example above with the pipe and `_` \ No newline at end of file From caf929918544a857f933d1a83e1aff9e055bca60 Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Tue, 28 Jul 2026 12:15:23 +0100 Subject: [PATCH 03/10] fill section about piping --- workflow_piping/index.qmd | 134 +++++++++++++++++++++++++++++++++++--- 1 file changed, 125 insertions(+), 9 deletions(-) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index 5bb7fda..dcd0677 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -5,6 +5,7 @@ author: "Etienne Bacher" date: "2026-07-27" categories: [r] difficulty: Beginner +toc: true format: html: default revealjs: @@ -18,25 +19,140 @@ editor: wrap: 72 --- -# Introduction +# Piping ## Example where we don't use the pipe -- assign each function output to an object +Let's say we want to keep the first 10 rows in the `mtcars` data, then keep the observations where `cyl >= 6`, and finally sort the remaining data by the `am` column. - - drawback: either create plenty of temp object names, or overwrite the same object again and again, in which case we have to be careful if we want to rename this object later. +Without using the pipe, we have two main approaches. -- alternative: use nested function calls +First, we can assign each function output to an object. +This object can either keep the same name and be overwritten at each step, or we can use a collection of temporary names: - - drawback: harder to create because we have to start from the most nested function and progressively go outwards +```{r} +# Same name gets overwritten +res <- head(mtcars, 10) +res <- subset(res, cyl >= 6) +res <- sort_by(res, ~ am) +res + +# Temporary names +tmp1 <- head(mtcars, 10) +tmp2 <- subset(tmp1, cyl >= 6) +res <- sort_by(tmp2, ~ am) +res +``` + +Each of these cases has drawbacks: + +- in the first case, if we want to rename `res` in the future then we must be careful to rename all its occurrences throughout the code. It also means that if, say, the call to `sort_by()` is wrong, then we must run the entire block again so that `res` is properly reset. Depending on the data size and operations to run, this can be very time-consuming. + +- in the second case, we pollute the global environment with potentially many temporary objects. Additionally, using a counter in a temporary name means that we need to update many names if we want to add an operation between the first and second step for instance. + + +The second approach isn't to define intermediate objects, but instead to run all those calls at once by nesting functions: + +```{r} +res <- sort_by(subset(head(mtcars, 10), cyl >= 6), ~ am) +res + +# Same but with different formatting +res <- sort_by( + subset( + head(mtcars, 10), + cyl >= 6 + ), + ~ am +) +res +``` + +We didn't define intermediate objects, but to read this code we now need to start from the innermost code (`head(mtcars, 10)`) and expand outwards. +This may hurt code readability. ## Introducing the pipe -[ add a quarto callout explaining the "history" of `%>%` and `|>` so that people know why `%>%` still appears in a lot of docs and examples ] +### Using "`|>`" + +The pipe is a way to chain operations in natural reading order by automatically passing the output of the left-hand side code to the right-hand side. +Using the pipe `|>` with the example above would give: + +```{r} +res <- head(mtcars, 10) |> + subset(cyl >= 6) |> + sort_by(~ am) +res +``` + +The pipe can be read as "and then": we keep the first 10 rows, *and then* we apply our filter, *and then* we sort the remaining data. + +We now have some code that doesn't require intermediate objects but is also easy to read because we see operations in the order in which they are executed. + +There are a couple of small restrictions in order to use `|>`: + +- this operator is available since R 4.1, released in 2021; +- it requires a proper function call in the right-hand side: + ```{r} + 1:3 |> mean() + ``` + ```{r} + #| error: true + 1:3 |> mean + ``` + + +### Using "`_`" + +In the code above, we could seamlessly chain operations because each of those functions take the input data as their first argument. + +This is not always the case. For instance, `grepl()` (which detects whether elements of a character vector match a specific pattern) takes the vector to check in second position: + +```{r} +let <- letters[1:6] +grepl("a|e", let) +``` + +Using `|>` without specifying the position of the input would lead to wrong code: + +```{r} +#| warning: true +let |> + grepl("a|e") +``` + +The code above is equivalent to `grepl(head(letters), "a|e")`, which is not what we want and is the cause of the warning. +We want to tell `grepl()` that the data we're passing via `|>` should end up in second position. +To do so, we can use `_`: + +```{r} +let |> + grepl("a|e", x = _) +``` + +Note that using `_` comes with a few restrictions: + +- this operator is available since R 4.2 (released in 2022); +- `_` must be used on a named argument, e.g. this fails: + ```{r} + #| error: true + let |> + grepl("a|e", _) + ``` +- a function call can contain only one `_`: + ```{r} + #| error: true + mtcars$drat |> + cor(x = _, y = _) + ``` + -- explain `|>`: how to use, constraints (R >= 4.1, RHS must be a function call with `()`) +### What about "`%>%`"? -- explain `_`: how to use, constraints (R >= 4.2, must be used as a named argument, can't appear more than once in the RHS) +You may have seen code that chains operations using `%>%`. +This `%>%` is part of the `magrittr` package. +It predates `|>` and was widely used in the [`tidyverse`](https://tidyverse.org/) until 2021, when `|>` was introduced in base R. -- rewrite the example above with the pipe and `_` \ No newline at end of file +For simple cases, `|>` and `%>%` behave identically. +We recommend using `|>` simply because it is always available in R and doesn't rely on an external package. \ No newline at end of file From ad15ca6c0f1aabaf299f38a07e43879f235c6c17 Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Tue, 28 Jul 2026 12:25:30 +0100 Subject: [PATCH 04/10] section about anonymous functions in pipes --- workflow_piping/index.qmd | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index dcd0677..c819eec 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -148,6 +148,21 @@ Note that using `_` comes with a few restrictions: ``` + +### Using anonymous functions in a piped chain + +So far, we have only used named functions provided in base R (though we could have used functions from other packages). +Sometimes, it is necessary to run custom code on the data without creating a dedicated new function for that. + +To do so, we need to wrap the call to the anonymous function in parenthesis and evaluate it with `()` at the end of its definition: + +```{r} +mtcars |> + subset(cyl == 4) |> + (function(d) lm(mpg ~ disp, data = d))() +``` + + ### What about "`%>%`"? You may have seen code that chains operations using `%>%`. From 430a964e557b132b349f5634cfd2066e78bb8ef8 Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Tue, 28 Jul 2026 12:26:45 +0100 Subject: [PATCH 05/10] less ws --- workflow_piping/index.qmd | 1 - 1 file changed, 1 deletion(-) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index c819eec..28ac229 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -148,7 +148,6 @@ Note that using `_` comes with a few restrictions: ``` - ### Using anonymous functions in a piped chain So far, we have only used named functions provided in base R (though we could have used functions from other packages). From 47cdce96bedee4d69794057a59917b6b37b5c21a Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Tue, 28 Jul 2026 12:43:15 +0100 Subject: [PATCH 06/10] typo --- workflow_piping/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index 28ac229..16cd6fa 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -122,7 +122,7 @@ let |> grepl("a|e") ``` -The code above is equivalent to `grepl(head(letters), "a|e")`, which is not what we want and is the cause of the warning. +The code above is equivalent to `grepl(let, "a|e")`, which is not what we want and is the cause of the warning. We want to tell `grepl()` that the data we're passing via `|>` should end up in second position. To do so, we can use `_`: From 0a9672fc455689432c66f629752bb848130a424a Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Mon, 10 Aug 2026 10:28:49 +0100 Subject: [PATCH 07/10] tweaks --- workflow_piping/index.qmd | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index 16cd6fa..8406f30 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -27,6 +27,8 @@ Let's say we want to keep the first 10 rows in the `mtcars` data, then keep the Without using the pipe, we have two main approaches. +### Intermediate objects + First, we can assign each function output to an object. This object can either keep the same name and be overwritten at each step, or we can use a collection of temporary names: @@ -51,13 +53,15 @@ Each of these cases has drawbacks: - in the second case, we pollute the global environment with potentially many temporary objects. Additionally, using a counter in a temporary name means that we need to update many names if we want to add an operation between the first and second step for instance. +### Nested calls + The second approach isn't to define intermediate objects, but instead to run all those calls at once by nesting functions: ```{r} res <- sort_by(subset(head(mtcars, 10), cyl >= 6), ~ am) res -# Same but with different formatting +# Same code with different formatting res <- sort_by( subset( head(mtcars, 10), @@ -165,8 +169,9 @@ mtcars |> ### What about "`%>%`"? You may have seen code that chains operations using `%>%`. -This `%>%` is part of the `magrittr` package. -It predates `|>` and was widely used in the [`tidyverse`](https://tidyverse.org/) until 2021, when `|>` was introduced in base R. +This `%>%` is also a pipe and it is provided by the `magrittr` package. +It predates `|>` and was widely used in the [`tidyverse`](https://tidyverse.org/). +Its design and popularity inspired the implementation of `|>` in base R in 2021. For simple cases, `|>` and `%>%` behave identically. We recommend using `|>` simply because it is always available in R and doesn't rely on an external package. \ No newline at end of file From 2b8725e6d26f14cdf3c5e2038817c8463477a5b9 Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Wed, 12 Aug 2026 09:47:13 +0100 Subject: [PATCH 08/10] address review comments --- workflow_piping/index.qmd | 75 ++++++++++++++++++++++++++++++--------- 1 file changed, 58 insertions(+), 17 deletions(-) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index 8406f30..9995c7a 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -23,10 +23,13 @@ editor: ## Example where we don't use the pipe -Let's say we want to keep the first 10 rows in the `mtcars` data, then keep the observations where `cyl >= 6`, and finally sort the remaining data by the `am` column. +Let's say we want to process the `mtcars` dataset as follows: -Without using the pipe, we have two main approaches. +* keep the first 10 rows +* keep the observations where `cyl >= 6` +* sort the remaining data by the `am` column. +We could do this in two different ways: by assigning intermediate output or by nesting function calls. ### Intermediate objects First, we can assign each function output to an object. @@ -92,19 +95,39 @@ res The pipe can be read as "and then": we keep the first 10 rows, *and then* we apply our filter, *and then* we sort the remaining data. +:::{.callout-note title="Assigning the output" collapse="true"} +While we have used `<-` to assign the output above, we could have used `->` to move the assignment to the end of the chain: + +```{r} +head(mtcars, 10) |> + subset(cyl >= 6) |> + sort_by(~ am) -> res + +# Equivalent: +head(mtcars, 10) |> + subset(cyl >= 6) |> + sort_by(~ am) -> + res +``` +::: + We now have some code that doesn't require intermediate objects but is also easy to read because we see operations in the order in which they are executed. -There are a couple of small restrictions in order to use `|>`: +The `|>` operator was introduced in R 4.1, released in 2021, meaning that you cannot use it in older versions of R. -- this operator is available since R 4.1, released in 2021; -- it requires a proper function call in the right-hand side: - ```{r} - 1:3 |> mean() - ``` - ```{r} - #| error: true - 1:3 |> mean - ``` +:::{.callout-note title="Error: The pipe operator requires a function call as RHS" collapse="true"} +Note that `|>` requires a function call on the right-hand side: + +```{r} +1:3 |> mean() +``` +```{r} +#| error: true +1:3 |> mean +``` + +This differs from the `magrittr` pipe, `%>%`. +::: ### Using "`_`" @@ -135,10 +158,19 @@ let |> grepl("a|e", x = _) ``` +Some operations don't have named arguments but still require specifying `_`, e.g. to extract a column and then compute its mean: + +```{r} +mtcars |> + _$drat |> + mean() +``` + + Note that using `_` comes with a few restrictions: - this operator is available since R 4.2 (released in 2022); -- `_` must be used on a named argument, e.g. this fails: +- `_` must be used on a named argument (except in some cases, such as `$` shown above). For example, this fails: ```{r} #| error: true let |> @@ -154,10 +186,19 @@ Note that using `_` comes with a few restrictions: ### Using anonymous functions in a piped chain -So far, we have only used named functions provided in base R (though we could have used functions from other packages). -Sometimes, it is necessary to run custom code on the data without creating a dedicated new function for that. +So far, we have only used functions provided in base R (though we could have used functions from other packages). +Sometimes, it is necessary to run custom code on the data without creating a dedicated new function for that, i.e. we want to use an *anonymous* function. +If you have used one of the `*apply()` functions before, then you may have used anonymous functions: + +```{r eval = FALSE} +# The function below is anonymous: it isn't assigned to anything, we +# just create it on the fly: +lapply(my_list, function(x) { + (x - mean(x) / sd(x)) +}) +``` -To do so, we need to wrap the call to the anonymous function in parenthesis and evaluate it with `()` at the end of its definition: +To use an anonymous function in a piped chain, we need to wrap it in parentheses and evaluate it with `()` at the end of its definition: ```{r} mtcars |> @@ -174,4 +215,4 @@ It predates `|>` and was widely used in the [`tidyverse`](https://tidyverse.org/ Its design and popularity inspired the implementation of `|>` in base R in 2021. For simple cases, `|>` and `%>%` behave identically. -We recommend using `|>` simply because it is always available in R and doesn't rely on an external package. \ No newline at end of file +We recommend using `|>` simply because it is always available in R and doesn't rely on an external package (and the [Tidyverse style guide](https://style.tidyverse.org/pipes.html) also recommends `|>`). \ No newline at end of file From c0b08004f32e9ee6d849bca394ae81552f02dd5d Mon Sep 17 00:00:00 2001 From: Etienne Bacher <52219252+etiennebacher@users.noreply.github.com> Date: Thu, 27 Aug 2026 14:37:14 +0100 Subject: [PATCH 09/10] Update workflow_piping/index.qmd Co-authored-by: William Gearty --- workflow_piping/index.qmd | 1 + 1 file changed, 1 insertion(+) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index 9995c7a..01ec941 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -30,6 +30,7 @@ Let's say we want to process the `mtcars` dataset as follows: * sort the remaining data by the `am` column. We could do this in two different ways: by assigning intermediate output or by nesting function calls. + ### Intermediate objects First, we can assign each function output to an object. From 58f16334a51326d72b9d512619b197fe885d0165 Mon Sep 17 00:00:00 2001 From: etiennebacher Date: Thu, 27 Aug 2026 14:43:41 +0100 Subject: [PATCH 10/10] address more comments --- workflow_piping/index.qmd | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/workflow_piping/index.qmd b/workflow_piping/index.qmd index 01ec941..6b9744b 100644 --- a/workflow_piping/index.qmd +++ b/workflow_piping/index.qmd @@ -34,7 +34,7 @@ We could do this in two different ways: by assigning intermediate output or by n ### Intermediate objects First, we can assign each function output to an object. -This object can either keep the same name and be overwritten at each step, or we can use a collection of temporary names: +This object can keep the same name and be overwritten at each step: ```{r} # Same name gets overwritten @@ -42,7 +42,12 @@ res <- head(mtcars, 10) res <- subset(res, cyl >= 6) res <- sort_by(res, ~ am) res +``` + +However, if we want to rename `res` in the future then we must be careful to rename all its occurrences throughout the code. It also means that if, say, the call to `sort_by()` is wrong, then we must run the entire block again so that `res` is properly reset. Depending on the data size and operations to run, this can be very time-consuming. +An alternative is to use a collection of temporary names: +```{r} # Temporary names tmp1 <- head(mtcars, 10) tmp2 <- subset(tmp1, cyl >= 6) @@ -50,11 +55,7 @@ res <- sort_by(tmp2, ~ am) res ``` -Each of these cases has drawbacks: - -- in the first case, if we want to rename `res` in the future then we must be careful to rename all its occurrences throughout the code. It also means that if, say, the call to `sort_by()` is wrong, then we must run the entire block again so that `res` is properly reset. Depending on the data size and operations to run, this can be very time-consuming. - -- in the second case, we pollute the global environment with potentially many temporary objects. Additionally, using a counter in a temporary name means that we need to update many names if we want to add an operation between the first and second step for instance. +But this is also not ideal because we pollute the global environment with potentially many temporary objects. Additionally, using a counter in a temporary name means that we need to update many names if we want to add an operation between the first and second step for instance. ### Nested calls