image-daemon

Authenticated loopback control endpoints for Lisp image daemons.

The library defines the wire protocol shared by a detached Lisp image and the terminals controlling it: bounded length-prefixed readable packet frames with reader evaluation disabled, capability tokens, timestamp-bearing session identifiers with legacy compatibility, and deadline-bounded loopback connection and call primitives. Load the optional runtime for private endpoint discovery, persistent request services, terminal relay ownership, bounded replay and attachment client loops.

Systems and package

• ASDF system: image-daemon
• Optional SBCL runtime system: image-daemon/runtime
• Optional SBCL evaluation endpoint system: image-daemon/eval
• Test system: image-daemon/tests
• Package: IMAGE-DAEMON

Install the locked dependencies and run all tests:

./script/bootstrap
./script/check

The control protocol

(image-daemon:daemon-call port token ':status)

(multiple-value-bind (socket stream)
    (image-daemon:daemon-connect port)
  (image-daemon:daemon-write-packet stream packet)
  (image-daemon:daemon-read-response stream ':attach))

Packets are proper lists written readably and framed by one decimal character count. Reading rejects reader evaluation, unbalanced payloads, and frames above *DAEMON-PACKET-CHARACTER-LIMIT*. Every required response is bounded by *DAEMON-CONNECT-TIMEOUT-SECONDS*. Request frames keep the historical :LOCALGROUP-REQUEST tag for compatibility with deployed endpoints.

Persistent services and attachments

Create an endpoint with DAEMON-RUNTIME-CREATE and a request callback. The default mode publishes a discovery record and requires a registry directory and identifier. Pass :PUBLISH-P NIL for an unpublished ephemeral runtime; it needs neither and keeps no registry pathname. After your own startup transaction, call DAEMON-RUNTIME-START. Stop and unpublish a published runtime with DAEMON-RUNTIME-STOP. The callback receives (runtime request &key socket stream) after protocol and capability validation; write the response with the packet API. You may subclass DAEMON-RUNTIME and specialize DAEMON-RUNTIME-REQUEST-VALID-P and DAEMON-RUNTIME-ERROR-RESPONSE for an application protocol.

Use DAEMON-REGISTRY-DISCOVER to inspect complete discovery records. Registry writers serialize with a directory file lock; publication retains live or uncertain ownership unless the capability matches. Reconciliation deletes only definitely dead records that still equal the inspected value. Shutdown likewise compares the complete record, preserving an intervening replacement.

Create a RELAY with RELAY-CREATE. Implement the TRANSPORT- protocol for your foreground transport, or specialize it on a relay subclass to synchronize your UI’s state. RELAY-ATTACH queues the handshake and replay before live output, allows multiple observers and one controller, and revokes the former controller on takeover. Read semantic packets with RELAY-READ-ATTACHMENT. Resizes run under the relay lock, so a host resize method should publish to its UI thread without reacquiring that lock. Hosts decide when input readers may pause and whether releasing a foreground terminal requires process handoff.

ATTACHMENT-CREATE owns an asynchronous output writer. Pending output and replay have independent character limits. On overflow, shut down socket I/O before abortive stream close and join the writer with a bounded wait. The client-side DAEMON-ATTACH-CLIENT-RUN accepts input-readiness, event-read and resize callbacks; supply its socket or a shutdown callback that unblocks concurrent reads. Pass :packet-function to handle host-specific control packets in the receiver thread. Return true from that callback to end the attachment. Attachment handshakes include :history-start and :history-position character positions for replaying output produced during a reconnect.

For application-specific output, subclass ATTACHMENT and specialize ATTACHMENT-PACKET-STRING. Construct with MAKE-INSTANCE to queue greeting and replay before calling ATTACHMENT-START. Supply :TRANSPORT-LOCK when sharing transport with an external owner. Specialize ATTACHMENT-RELEASE-TRANSPORT for borrowed transport: call ATTACHMENT-SHUTDOWN to unblock I/O, remove membership idempotently, and leave stream closure to its owner. Release methods run outside the queue lock on every writer exit and may run repeatedly or concurrently.

ATTACHMENT-TRY-SEND returns :ACCEPTED, :CLOSED or :OVERFLOW without synchronous transport cleanup. Overflow discards pending output and wakes the writer; explicitly close a deferred attachment after overflow. ATTACHMENT-SEND retains its Boolean API and releases transport on overflow. ATTACHMENT-SEAL appends one final packet beyond the ordinary capacity and closes admission without starting or reaping a writer. ATTACHMENT-FINISH seals, starts a deferred drain, reaps with a bounded deadline and releases transport. An unread peer can lose pending output. Reaping failures propagate and retain the writer for retry.

When the hosted application is about to exit with a status its launcher acts on, call RELAY-FINISH with that status and an optional message. Every client receives it after the output already queued for it, and DAEMON-ATTACH-CLIENT-RUN returns it as (:STATUS STATUS :MESSAGE MESSAGE), so a relaying front process can exit exactly as a foreground application would have. Any other end returns NIL.

Process handoff

A process that hands its work to a detached replacement does it through a private ticket. handoff-ticket-pathname names one beneath a directory and handoff-ticket-write publishes the host’s record, which carries :state and :token. The replacement claims the ticket with handoff-ticket-claim, a rename that fails once the launcher has cancelled, and keeps checking handoff-ticket-owned-record while it starts. The launcher cancels with handoff-ticket-cancel, a rename the replacement then sees. Every state lives in a sibling named after the ticket (handoff-ticket-sibling), so handoff-ticket-delete removes the whole family.

handoff-launch-supervised starts the replacement behind a Bash supervisor that creates its process group, records the group identifier beside the ticket (handoff-ticket-launcher-pid), and only then lets the launcher run, so a failed handoff can always find what to stop. handoff-ticket-replacement-pid names the claimant. daemon-endpoint-status accepts a registry record only when a different process serves it with the expected token and answers an authenticated :status request, and daemon-wait-until polls such a check up to a deadline.

Evaluation endpoints

image-daemon/eval lets trusted local programs evaluate Common Lisp in the running image. eval-endpoint-create validates the settings: a private Unix socket or an IPv4 loopback TCP address, a token file, the package sources are read and evaluated in, and bounds on frames, sources, output, queued requests, clients and time. eval-endpoint-start listens and starts one accept thread and one serial evaluator; eval-endpoint-stop closes everything within a deadline and signals :quiesce naming any thread that stayed alive.

Every frame is a four-octet big-endian length followed by that many UTF-8 octets holding one S-expression. The length is checked before the payload is read, and the payload is parsed by a closed sexp-config grammar, so no frame, authenticated or not, interns a symbol or nests without bound. A connection starts with (:challenge :version 1 :algorithm :hmac-sha-256 :nonce HEX) and must answer (:authenticate :proof HEX), the HMAC-SHA-256 of the nonce keyed by the token file’s content. The endpoint reads the token only while checking a proof, refuses a token file that is not a private regular file, wipes every secret octet, and retains only the pathname. Authenticated clients then send (:evaluate :source TEXT) one at a time and receive an :evaluation-result with bounded values, captured output, and a :condition or :timeout status for failures, including debugger entry.

A Unix endpoint refuses an active socket and any object it does not own; a stale socket is renamed aside and removed only while it is still the socket that was inspected. eval-connect and eval-call are a complete client, and eval-proof computes a proof for clients written elsewhere.

(let ((endpoint (image-daemon:eval-endpoint-create
                 :transport ':unix
                 :unix-pathname #p"/run/user/1000/app/eval.sock"
                 :token-pathname #p"/home/me/.config/app/eval.token"
                 :package "MY-APP")))
  (image-daemon:eval-endpoint-start endpoint))

Existing-loop deadline callbacks

daemon-runtime-set-timer installs paired callbacks on an existing runtime:

(image-daemon:daemon-runtime-set-timer
 runtime
 :next-deadline-function (lambda (runtime)
                           (declare (ignore runtime))
                           (host-next-universal-time-deadline))
 :wake-function (lambda (runtime)
                  (declare (ignore runtime))
                  (host-materialize-due-work)))

The next-deadline callback returns a nonnegative universal-time integer or NIL. The wake callback runs on the existing accept-loop thread outside the runtime lock and must advance or clear a due deadline before returning. Listener readiness and deadlines share the same blocking wait, without polling or another execution thread. Keep callbacks bounded and enqueue ordinary host work rather than performing it on the accept thread. Timer errors reach the existing runtime error callback.

Configure before startup or replace live. Clear both callbacks with (daemon-runtime-set-timer runtime). After changing a later host deadline, call daemon-runtime-notify; its authenticated internal :runtime-notify operation wakes the listener without invoking the application’s request callback. It returns T after acknowledgement, or NIL before start/after stopping. Hosts can also notify a remote runtime through ordinary daemon-call with :runtime-notify. Deliver observable events through the host’s normal authenticated callback, update its state and notify if the next deadline changed.

Bounded message operations (optional)

Load image-daemon/messages. Create an endpoint with daemon-message-endpoint-create and the usual :directory, :identifier and :publish-p settings, plus :dispatcher. Start/stop and discover it with the existing daemon runtime and private registry API. Defaults are 65,536 request and response characters, 32 concurrent clients, and a 2-second request timeout. :character-limit, :client-limit and :request-timeout replace those bounds. Admission is bounded before frame parsing, including stalled unauthenticated clients.

The dispatcher receives (envelope &key cancelled-p) and returns one portable result. The envelope is (:action KEYWORD :id STRING :sender STRING :receiver STRING :payload VALUE). IDs are stable host-selected strings of 1..256 characters. Payloads/results contain proper lists, strings, rational numbers, keywords, T and NIL. Envelopes are framed by the existing bounded readable daemon protocol, retain its capability authentication and correlate replies by ID.

(image-daemon:daemon-message-call
 (cdr (first (image-daemon:daemon-registry-discover directory)))
 :action ':send :id "m1" :sender "a" :receiver "b"
 :payload '(:text "ready") :timeout 2)

daemon-message-call takes the tagged private discovery record, not a display record or a bare plist. Optional :cancelled-p cancels local waiting; semantic cancellation/acknowledgement is a separate dispatcher action. Runtime shutdown is observable through the dispatcher’s cancellation predicate. Keep capability-bearing records private and out of logs or saved application payloads.

To compose an existing service, route authenticated :message operations to daemon-message-handle-request runtime request :stream stream :dispatcher function, with optional :character-limit and :request-timeout. The existing service’s pre-dispatch frame policy applies; specialize daemon-runtime-read-request when a smaller bound is required. daemon-runtime-client-limit is the corresponding optional concurrent-client protocol.

The host owns authorization of sender/receiver identities, stable-ID retention, mailbox/wakeup semantics and durable acknowledgement. No job-pool or application package is required. Request timeout, cancellation, disconnect or lost replies can follow committed effects. Resolve an uncertain outcome through the SAME stable ID, rather than assuming rejection and generating another ID. Failures signal daemon-message-error (a daemon-error), with daemon-message-error-reason such as :payload-limit, a dispatcher-supplied rejection, :cancelled or :uncertain.

Run optional actual subprocess discovery/transport and timer checks with:

(asdf:test-system :image-daemon/message-tests)

Failures

Failures signal DAEMON-ERROR carrying a message, the failing operation, and the session identifier when known. Hosts may substitute a subclass through *DAEMON-ERROR-CLASS* to join their own condition hierarchy. Evaluation endpoints signal EVAL-ENDPOINT-ERROR, a DAEMON-ERROR that adds a non-secret reason keyword and, for :quiesce, the names of the threads that did not stop.

Thread stops use one deadline across graceful exit and forced termination. If unwind cleanup cannot finish, handle DAEMON-ERROR with operation :STOP-THREAD. Recover the target with SB-THREAD:THREAD-ERROR-THREAD on DAEMON-ERROR-CAUSE, release the blocked cleanup, and retry DAEMON-STOP-THREAD. Attachment and runtime owners retain their thread references when a stop fails.

Development

./script/bootstrap
./script/check
./script/check --fix

./script/check runs the tests, then checks the source with cheque as cheque.sexp configures it. --fix skips the tests and repairs what cheque can repair. cheque and its tools are development dependencies: checks/qlfile lists them, and the check installs them on first use into ~/.cache/cheque/tools/.

License

COLL-Attribution. See LICENSE.lisp.