Add resource usage measurement injection to CWL interpreter by avnig05 · Pull Request #5526 · DataBiosphere/toil

avnig05 · 2026-05-28T19:46:42Z

CWL and WDL have a shared runtime injection module that samples in-container CPU/memory usage and reports it back to Toil via runtime message files.

Changelog Entry

To be copied to the draft changelog by merger:

toil-cwl-runner --stats now records resource usage from Docker containers so toil stats can read it.
(Edited by @adamnovak)

Reviewer Checklist

Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
- If it is coming from an external repo, make sure to pull it in for CI with:
```
contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
```
- If there is no associated issue, create one.
Read through the code changes. Make sure that it doesn't have:
- Addition of trailing whitespace.
- New variable or member names in camelCase that want to be in snake_case.
- New functions without type hints.
- New functions or classes without informative docstrings.
- Changes to semantics not reflected in the relevant docstrings.
- New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
- New features without tests.
Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
Finish the review with an overall description of your opinion.

Merger Checklist

Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
Merge with the Github "Squash and merge" feature.
- If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
Copy its recommended changelog entry to the Draft Changelog.
Append the issue number in parentheses to the changelog entry.

adamnovak · 2026-05-28T20:37:26Z

@avnig05 It looks like this isn't passing type checking; if you make mypy you should be able to reproduce the problems:
https://ucsc-ci.com/databiosphere/toil/-/jobs/110094#L1195

It looks like mostly you need casts or # type: ignore comments at some places where you're using a fake object or looking in one of these Expando objects that are fundamentally dynamically-typed. But it looks like you might not be importing ResourceMonitor from the place where it's actually defined?

adamnovak

This looks like it should work for the Docker container case.

I think for the Singularity/CWL case, it might be double-counting the resource usage, because it will be counted via having a child process and counted again via the command injection.

I think it might be possible to simplify the hooking into the CWL code by overriding another method in ToilCommandLineTool that gets to run after the container finishes, which would let us avoid needing ToilContainerCommandLineJob and friends and the machinery to attach them.

I think the PR should be rebased to not be on top of the pre-squash commits from #5512.

adamnovak · 2026-05-28T20:41:08Z

+    def _file_mounts_from_pathmapper(
+        job: ContainerCommandLineJob,
+    ) -> list[tuple[str, str]]:


This needs a docstring to at least explain which item in the result tuples is which.

adamnovak · 2026-05-28T21:04:12Z

+class ToilContainerCommandLineJob(ContainerCommandLineJob):
+    """Container job that collects resource stats from injected in-container code."""
+
+    def _execute(
+        self,
+        runtime: list[str],
+        env: MutableMapping[str, str],
+        runtimeContext: cwltool.context.RuntimeContext,
+        monitor_function: Callable[["subprocess.Popen[str]"], None] | None = None,
+    ) -> None:
+        super()._execute(runtime, env, runtimeContext, monitor_function)
+        handle_injection_messages_from_outdir(self.outdir)
+
+
+class ToilDockerCommandLineJob(ToilContainerCommandLineJob, DockerCommandLineJob):
+    """Docker container job with Toil runtime injection support."""
+
+
+class ToilPodmanCommandLineJob(ToilContainerCommandLineJob, PodmanCommandLineJob):
+    """Podman container job with Toil runtime injection support."""
+
+
+class ToilSingularityCommandLineJob(ToilContainerCommandLineJob, SingularityCommandLineJob):
+    """Singularity container job with Toil runtime injection support."""
+


This approach of hooking the ContainerCommandLineJob in addition to the CommandLineTool is a little awkward. We have to have all these extra classes for each container system, and we already don't have one for UDockerCommandLineJob and it's hard to tell whether that's because we don't need one. And the hook here only works in concert with the hook that ToilCommandLineTool applies, so we end up with one feature spread out over several classes.

Instead of hooking ContainerCommandLineJob._execute(), did you consider hooking CommandLineTool.collect_output_ports() instead? It looks like that has access to the outdir, and it gets sent into _execute() and called inside there after the container has run. And if we did it that way we could keep all the hook logic together inside ToilCommandLineTool and not need to worry as much about the container type.

adamnovak · 2026-05-28T21:06:10Z

+        for job in super().job(job_order, output_callbacks, runtimeContext):
+            if isinstance(job, ContainerCommandLineJob) and self._uses_container(
+                runtimeContext
+            ):
+                file_mounts = self._file_mounts_from_pathmapper(job)
+                script = command_line_to_shell_script(job.command_line)
+                script = add_injections(script, file_mounts)
+                job.command_line = shell_script_to_command_line(script)


Is this going to inject the monitoring logic even when we're using Singularity (or I think UDocker?) where we should already see CPU and memory usage in the stats because it happens under a child process of Toil? Will that lead to double-counting of CPU usage?

adamnovak · 2026-05-28T21:10:00Z

+###
+# Runtime code injection system
+# When a workflow steps runs inside a container, the Toil worker process on the host
+# often cannot see how much CPU and RAM that step actually used. This system allows


The word often is doing some vigorous handwaving here, and trying to hide that this text does not convey an understanding of how or why (or, more importantly for avoiding double-counting of resource usage, when) this will be the case.

adamnovak · 2026-05-28T21:11:47Z

+    present, and a flat argv list otherwise.
+    """
+    if (
+        len(command_line) >= 3


This might want to be == or we'll be able to throw away arguments.

adamnovak · 2026-05-28T21:32:25Z

+                    "--outdir",
+                    str(tmp_path / "output_dir"),
+                    str(cwl_file),
+                    str(inputs_file),
+                    "--jobStore",
+                    str(job_store),
+                    "--stats",


There's nothing in here to make Docker be used, but I looked at the toil-cwl-runner options and it has --singularity, --podman, and --no-container, but not a --docker. So I guess Docker is the default container engine. Maybe we need a comment here to remind us that the test depends on that?

adamnovak · 2026-05-28T21:35:37Z

+    # Include malformed and partial lines to exercise parser robustness.
+    message_file.write_text(
+        "CPU\t1000000\n"
+        "Memory\t1024\n"
+        "CPU\t4000000\n"
+        "Memory\t2048\n"
+        "Odd\tline\n"
+        "CPU\t5000000",
+        encoding="utf-8",
+    )


One thing that's noticeably absent here is a line with the wrong number of fields, and in particular too few fields.

adamnovak · 2026-05-28T21:42:24Z

+    from toil.lib import interpreter
+
+    message_file = tmp_path / "resources.tsv"
+    # Include malformed and partial lines to exercise parser robustness.


A some of what's being tested here (like the fact that final lines without a terminating newline are ignored) seems to be based on the implementation code, rather than anything that the docstring actually promises will be true about the function under test.

If it's an important enough point about the function's behavior that it's worth testing, it's probably worth documenting in the docstring. Otherwise, the next person to touch the function is going to tinker with the internals in a way that doesn't appear to change anything about what the function promises, and then be hit with test failures because there are other secret things that need to be true about the function.

adamnovak · 2026-05-28T21:43:16Z

+    job = object.__new__(cwltoil.ToilDockerCommandLineJob)
+    job.command_line = ["echo", "hello"]
+    job.pathmapper = FakePathMapper(FakeMapperEntry(str(host_input), "/work/input.txt"))
+


There aren't any actual assertions in this test, so I don't think it's genuinely testing what its docstring claims it is testing.

adamnovak · 2026-05-28T21:50:56Z

I think these are leftover changes from #5512. It looks like your new commit c4c69a4 is on top of the commits from the branch for that PR, and not on top of the squashed commit that was created when the PR was merged.

I would recommend rebasing onto the current mainline Toil, and keeping just the commit about this feature in the branch for this PR:

Be on the branch for this PR.

Do git fetch origin (assuming your origin remote points to this Github project)

Do git rebase -i origin/master

In the editor, delete all the lines for the commits from the old feature, and only keep the line for the commit that's about the new feature.

Save and quit

Git should rewrite your history so now your issues/5477-cpu-usage-docker-cwl branch just has the new commit for this feature.

Do a git push origin issues/5477-cpu-usage-docker-cwl -f to force-push your rewritten history to Github and update the PR.

adamnovak · 2026-06-11T20:52:02Z

@avnig05 Since you're gone for the summer, I'm going to take on landing this myself.

…n system

I made the changes I wanted.

…l user argument lists

… supposed to be

mr-c

How does this interact with the built-in resource monitoring in cwltool?

https://github.com/common-workflow-language/cwltool/blob/1bf74499ca1c4a5f98e7cffb0ad4aa89aa98cb9e/cwltool/job.py#L522

https://github.com/common-workflow-language/cwltool/blob/1bf74499ca1c4a5f98e7cffb0ad4aa89aa98cb9e/cwltool/job.py#L861

mr-c · 2026-07-02T22:45:05Z

+
+    The resulting command required Bash to be available.
+    """
+    return ["bash", "-c", script]


Some containers and hosts don't have bash

We only use this when doing the injection, which only has to worry about containers, but indeed we'd run into a problem if the container doesn't have Bash.

The script we inject contains bash-isms, and uses head, cut, and awk, which are not shell built-ins and would have to be available inside the container for the monitoring to actually work.

We could translate the script back to /bin/sh (maybe also pulling in a dependency on something else for the math), and add some tests for what happens if a user container doesn't ship standard tools an only ships their user binary (the monitoring can't work, but we need it to fail in a safe way). Or we could figure out how to get something smart enough to do all this injected into the container (maybe a Linux binary we mount?). Or we could try and figure out how to do the monitoring outside the container, but that probably would require changes in both cwltool and miniwdl.

adamnovak · 2026-07-06T20:13:54Z

How does this interact with the built-in resource monitoring in cwltool?

This duplicates the functionality of the cwltool built-in memory monitoring, and adds CPU usage monitoring.

We could call out to the cwltool memory usage monitoring and only use our own code for CPU usage monitoring, but since Toil needs code for both on the WDL side it seemed simplest to just monitor both the same way on the CWL side as well.

If cwltool grew built-in CPU monitoring we could stop trying to tack on our own logic and use that instead.

adamnovak · 2026-07-06T20:22:43Z

For some discussion about the design on the WDL side, see chanzuckerberg/miniwdl#672. Mike Lin wrote up a plugin for MiniWDL demonstrating within-container monitoring, but that wasn't really quite published as installable and also ran the statistics in-band through the user-visible standard error stream, so I implemented it again with the same basic approach in Toil. If we really want to cut the whole command injection concept from Toil, we'd need to build out CPU monitoring in cwltool, and spruce up that plugin in MiniWDL so Toil could depend on it instead.

adamnovak · 2026-07-06T21:55:44Z

The CWL conformance tests are failing on initial_workdir_empty_writable_docker, which might indicate that we're not actually allowed to hide the communications channel for the resource monitoring inside the working directory.

…sage-docker-cwl

Accoridng to https://github.com/torvalds/linux/blob/8cdeaa50eae8dad34885515f62559ee83e7e8dda/kernel/sched/cpuacct.c#L262 we have user and system fields here, and actually probably we need to be including both.

adamnovak · 2026-07-06T23:15:30Z

OK, I have code to fix the need for Bash or other utilities in my Toil fork, but I still haven't worked out how to inject another mount from where I'm working.

…wl' into issues/5477-cpu-usage-docker-cwl

…-the-right-exit-code WDL test more

adamnovak · 2026-07-09T15:34:30Z

The remaining problem here is the docker_entrypoint CWL test.

If I need to inject a script into the container, I need to figure out what the entrypoint is, put it into the script, and override the container's entrypoint with the shell.

(Another complication is that it's entirely possible to make a Docker container without any shell at all, just with the application binary. We might end up having to stuff a busybox in there. At which point why am I not sending something more useful? And will user code care if it isn't PID 1?)

mr-c · 2026-07-09T17:37:07Z

For containers you can monitor memory and CPU usage outside the container

adamnovak · 2026-07-09T18:36:55Z

For containers you can monitor memory and CPU usage outside the container

Only if I do my own integration of the point-in-time CPU percentage from docker stats. The total CPU usage in core-seconds is not available except from the Docker API (see https://stackoverflow.com/questions/30271942/get-docker-container-cpu-usage-as-percentage and https://github.com/TheClimateCorporation/docker/blob/master/docs/reference/commandline/stats.md#formatting). And I don't think that the stats samples are very timestamped the way cwltool'sdocker_monitor collects them now: https://github.com/common-workflow-language/cwltool/blob/1bf74499ca1c4a5f98e7cffb0ad4aa89aa98cb9e/cwltool/job.py#L906-L911. So integrating over time would be hard.

I guess I could try and talk to the Docker API from the Toil CWL interpreter layer. But I don't like having that contact surface (cwltool itself never seems to use the Docker API), and then I probably end up with toil-and-cwl-specific code to do it, that I think wouldn't be able to be used on the WDL side because that's all Docker Swarm.

If cwltool-side stuff gains support for measuring (or back-calculating) core-seconds used, I could call into it for CWL, and that would be way better than this. But I'm trying to avoid getting into the existing monitoring code. (I'm also trying to put this project down because I am close to being out of available time to work on it, and I'm reluctant to throw out the whole approach I'd initially set Avni on and not merge any of her work.)

mr-c · 2026-07-10T10:03:47Z

+class ToilDockerCommandLineJob(DockerCommandLineJob):
+    """Container job that collects resource stats from injected in-container code."""
+
+    def _execute(


This sub-class of DockerCommandLineJob (itself a sub-class of ContainerCommandLineJob) doesn't override ContainerCommandLineJob.run() which does its own process monitoring: https://github.com/common-workflow-language/cwltool/blob/1bf74499ca1c4a5f98e7cffb0ad4aa89aa98cb9e/cwltool/job.py#L849-L858

avnig05 requested a review from adamnovak May 28, 2026 19:46

adamnovak previously requested changes May 28, 2026

View reviewed changes

Avni Gandhi and others added 8 commits July 1, 2026 13:32

add injected container cpu/memory accounting for cwl and wdl

cd214b2

Change output hook and avoid hooking container types that don't need it

62e252b

Fix type checking

8da4efd

Re-document interpreter.py

52d319e

Address interpreter.py review comments

65d3078

Note dependency on default runner

cac8297

Remove incomplete or unwanted tests

4cf943a

Make interpreter.py able to be about more than just the code injectio…

ef5d547

…n system

adamnovak force-pushed the issues/5477-cpu-usage-docker-cwl branch from 02594d1 to ef5d547 Compare July 1, 2026 20:19

adamnovak changed the title ~~Issues/5477 cpu usage docker cwl~~ Add resource usage measurement injection to CWL interpreter Jul 1, 2026

adamnovak added 3 commits July 1, 2026 19:40

Stop requiring a fixed Bash path and stop unpacking what could be rea…

79401f3

…l user argument lists

Add Anthropic Claude's idea of what waste_cpu_memory.cwl probably was…

02dc3a6

… supposed to be

Merge branch 'master' into issues/5477-cpu-usage-docker-cwl

15d11fd

mr-c reviewed Jul 2, 2026

View reviewed changes

adamnovak added 6 commits July 6, 2026 17:56

Merge remote-tracking branch 'upstream/master' into issues/5477-cpu-u…

a204780

…sage-docker-cwl

POSIX-ify injected shell code

3a9bccf

Stop assuming stat

95868c0

Read cpu stats the same way for cgroups v1 as v2

b36cda7

Accoridng to https://github.com/torvalds/linux/blob/8cdeaa50eae8dad34885515f62559ee83e7e8dda/kernel/sched/cpuacct.c#L262 we have user and system fields here, and actually probably we need to be including both.

Mechanically translate awk to shell

2eebbfc

Complain about how we need to add a mount somehow

f17e682

Hide the usage data in the temp directory

f5b868e

adamnovak added 4 commits July 6, 2026 19:42

Merge remote-tracking branch 'upstream/issues/5477-cpu-usage-docker-c…

df4af80

…wl' into issues/5477-cpu-usage-docker-cwl

Make sure to stop the background monitoring process in the container

a3f3cbb

Make bucket creation handle retry better if the bucket got created

17916a0

Pass through Interrupted exit codes in hopes of passing the fail-with…

4158f87

…-the-right-exit-code WDL test more

Move injection back into job class but just one this time

e5e2390

adamnovak merged commit 53f018b into master Jul 9, 2026
3 checks passed

adamnovak deleted the issues/5477-cpu-usage-docker-cwl branch July 9, 2026 21:02

adamnovak mentioned this pull request Jul 9, 2026

Record CPU usage of Docker containers under CWL #5477

Closed

mr-c reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

avnig05 commented May 28, 2026 • edited by adamnovak Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog Entry

Reviewer Checklist

Merger Checklist

Uh oh!

adamnovak commented May 28, 2026

Uh oh!

adamnovak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamnovak commented Jun 11, 2026

Uh oh!

mr-c left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamnovak commented Jul 6, 2026

Uh oh!

adamnovak commented Jul 6, 2026

Uh oh!

adamnovak commented Jul 6, 2026

Uh oh!

adamnovak commented Jul 6, 2026

Uh oh!

adamnovak commented Jul 9, 2026

Uh oh!

mr-c commented Jul 9, 2026

Uh oh!

adamnovak commented Jul 9, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

avnig05 commented May 28, 2026 •

edited by adamnovak

Loading