Skip to content

fix(monitor): drain console PTY master in background mode - #247

Merged
ravindu644 merged 1 commit into
ravindu644:devfrom
seriaTvT:fix/background-console-drain
Jul 23, 2026
Merged

fix(monitor): drain console PTY master in background mode#247
ravindu644 merged 1 commit into
ravindu644:devfrom
seriaTvT:fix/background-console-drain

Conversation

@seriaTvT

Copy link
Copy Markdown
Contributor

Problem

In background mode (foreground=0), nothing ever reads the container's console PTY master. The container's /dev/console is the matching slave. Once the container's init writes more than the kernel's PTY buffer (~64 KiB) to /dev/console, the write blocks forever in n_tty_write, and because the buffer never drains, every subsequent console write blocks too. The container wedges permanently and can only be killed with SIGKILL.

This is independent of guest distro and any sufficiently chatty init triggers it. It reproduces trivially with a bare dd. All early boot/console output is also lost in background mode, since it dies in the undrained master, so there's no way to see why a container failed to come up.

Root cause

  • Foreground path drains the master via console_monitor_loop (src/console.c), called from src/container.c.
  • Background path never calls it — src/container.c just polls for the boot marker and returns.
  • The monitor (src/monitor.c) inherits the master fd but only redirects its own stdio to /dev/null, then sits in its waitpid/virtualize heartbeat loop. It never reads the master.

Fix

The monitor already runs a 500ms heartbeat poll() loop. This adds the console master to that poll set (in background mode only) and drains it whenever readable — poll() wakes immediately on readability, so draining is effectively real-time. Drained bytes are appended to a per-container console log (Logs/<name>/console, capped at 2 MiB), which also recovers the previously-lost background boot logs. Draining continues even if the log write fails — keeping the buffer empty is the load-bearing part.

Foreground mode is untouched. Change is confined to src/monitor.c.

Terminal output

Before the fix — write 1 MiB to a background container's console, it never returns:

# dd if=/dev/zero of=/dev/pts/24 bs=1024 count=1024
(hangs forever)

# cat /proc/<dd-pid>/stack
[<...>] wait_woken
[<...>] n_tty_write+0x36c/0x444
[<...>] tty_write+0x23c/0x2f0
[<...>] do_iter_write / vfs_writev / do_writev / SyS_writev

After the fix — same write, on both test devices:

# kernel 4.14
monitor=495328  console_slave=/dev/pts/24  wchan=do_sys_poll
RESULT: COMPLETED in 100ms -> DRAIN WORKS, no deadlock
console log size=1056426 bytes

# kernel 6.12
monitor=190859  console_slave=/dev/pts/1  wchan=do_sys_poll
RESULT: COMPLETED in 51ms -> DRAIN WORKS, no deadlock
console log size=1060785 bytes

Tested environments

  • Android/arm64, kernel 4.14.357, MediaTek, LineageOS 23.2, background-mode Ubuntu container.
  • Android/arm64, kernel 6.12.23, Qualcomm, Xiaomi HyperOS 3, background-mode Ubuntu container.

Same result on both — rules out this being specific to an old/vendor kernel.

No regressions

  • Foreground mode path is untouched (no changes to console_monitor_loop or its callers).
  • Known-good background containers on both devices started, ran, and stopped (including a normal graceful shutdown) with no change in behavior other than the
    fix itself.

Notes

  • The buffer limit is the kernel PTY/tty default (~64 KiB); the exact figure doesn't matter. Any bound is exceeded by a sufficiently chatty boot, and once full it never drains today.
  • The console log is drained continuously and self-truncates at 2 MiB, so a runaway console producer can't fill the disk.
  • This bug was originally identified through Claude, then reproduced and verified on real hardware (both devices above) using the steps in this PR.

The monitor never read the console PTY master in background mode, only
foreground did (via console_monitor_loop). Once a container's init wrote
past the ~64 KiB PTY buffer, /dev/console blocked forever in
n_tty_write and the whole container wedged permanently.

Add the master to the monitor's existing heartbeat poll loop and drain
it continuously, appending output to a per-container console log. This
both fixes the deadlock and recovers previously-lost background boot
logs.
@ravindu644
ravindu644 changed the base branch from main to dev July 23, 2026 00:43
@ravindu644
ravindu644 merged commit 23a058d into ravindu644:dev Jul 23, 2026
2 checks passed
ravindu644 pushed a commit that referenced this pull request Jul 26, 2026
The monitor never read the console PTY master in background mode, only
foreground did (via console_monitor_loop). Once a container's init wrote
past the ~64 KiB PTY buffer, /dev/console blocked forever in
n_tty_write and the whole container wedged permanently.

Add the master to the monitor's existing heartbeat poll loop and drain
it continuously, appending output to a per-container console log. This
both fixes the deadlock and recovers previously-lost background boot
logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants