Commit Graph
18 Commits
Author SHA1 Message Date
jif 6bae551093 Bound managed app-server and updater stderr logging (#55856)
## Why

Managed stderr logs previously retained a tail only before a new launch, allowing logs to grow during a long-running process. Direct stderr writes could also block lifecycle operations on disk I/O.

## What changed

- Route managed app-server and updater logs through a shared nonblocking worker with a bounded queue. Drop records when the queue is full or a record exceeds 16 KiB, and avoid waiting for the worker at shutdown.
- Prune live logs when they exceed 1 MiB, preserving the latest 256 KiB in `.log.previous` before truncating the current file. Keep inherited stderr handles usable after truncation.
- Pause predecessor pruning during Windows updater handoff and resume it after a failed handoff only when ownership has been restored.
- Keep ordinary stderr output when managed logging has not been initialized.

## Testing

Add regression tests for stalled workers and queue overflow, tail retention, inherited append and nonappend handles, and pruning exclusion during updater handoff.

GitOrigin-RevId: b6301962b1180fa6cdb231a56edce2429dc20b54
2026-10-21 10:31:12 +00:00
Eric Traut 2f05c0a71f Fix daemon updater socket binding and bootstrap readiness on Unix (#53845)
## Why

The updater bound its socket directly beneath `CODEX_HOME`, which can exceed Unix socket path limits. Bootstrap could also report success before the updater's manual-update socket was ready.

## What changed

- Reuse the control socket's protected binding and cleanup guard for the updater, with a symlink at the advertised socket path.
- Remove stale updater socket entries before startup while preserving regular files.
- Wait up to 30 seconds for supported automatic updaters to publish their socket, without connecting as a readiness probe. Report updater exit or timeout with stderr log context.
- Preserve compatibility with older updaters that lack the manual-update command.

## Testing

Add Unix integration tests for updater readiness with a long `CODEX_HOME` path and bootstrap failure when a regular file blocks the updater socket, verifying that the file is preserved and the daemon remains running.

GitOrigin-RevId: 57fdcbe4e6907ece68ba66178cf81f258527560c
2026-10-14 02:51:53 +00:00
Eric Traut 0960e379a5 Prune unused app-server daemon releases safely (#53827)
## What changed

- Record the current and previous dedicated daemon selections in `selection-history`, bound to the selection link's identity. Preserve that history when an install leaves the selection unchanged.
- Prune unused releases after package preparation and scheduled update processing, including updates that require no restart. Retain the current and previous selections, packages used by the daemon or updater, and the calling executable's package.
- Require lifecycle and installer locks, valid selection history, and verified package boundaries before cleanup. Skip unsafe or unrecognized packages and retire releases before best-effort deletion so partial failures are not retried.
- Resolve the updater's actual executable identity after an executable handoff that preserves its PID.

## Testing

Extend coverage for stale selection history, retention of the last two selections, packages containing external symlinks, partial deletion and reinstall, cleanup without restarting the daemon, and repeated installs preserving selection history.

GitOrigin-RevId: 19a04db1cc356a2219f2afb137b9525e249ab6fb
2026-10-14 01:49:56 +00:00
Eric Traut 1a169eda11 Distinguish daemon release identity from executable contents (#50559)
## Why

Releases can contain different resources while shipping identical executable bytes. The daemon must recognize those release changes, while a stale updater should be able to hand off without restarting an already-current daemon.

## What changed

- Include a digest of the resolved executable path in `ExecutableIdentity` to distinguish release generations with identical binaries.
- Keep content-only comparisons for package preparation and manual update reporting.
- Check the running daemon's identity and version before restarting during updater adoption, manual updates, and production restoration. Preserve scheduled retries when the daemon is not ready.

## Testing

Add coverage for path-sensitive identity, scheduled no-op updates that preserve the daemon, and resource-only release updates that restart it despite identical executable bytes.

GitOrigin-RevId: bbea06cf502abb2251e70d4ba4b422cae867116c
2026-10-03 07:17:20 +00:00
Eric Traut 55922b7984 Include installer stderr in daemon update failures (#50499)
## Why

Daemon updates discard installer stderr, leaving failed installs with only an exit status and no installer diagnostics.

## What changed

Capture the last 2 KiB of installer stderr and append it to errors when the installer exits unsuccessfully. Drain stderr concurrently and wait at most one second after failure so subprocesses holding the pipe open cannot delay error reporting indefinitely. Keep captured details within the manual update response limit, and configure PowerShell output as UTF-8 to preserve non-ASCII diagnostics.

## Testing

Add coverage for stderr tail truncation, bounded draining when a subprocess keeps stderr open, and manual update responses containing heavily escaped diagnostics. Extend the Windows installer test to check non-ASCII error text.

GitOrigin-RevId: d0da19b233006f97770d3f7f5f057e5e98fd2913
2026-10-03 02:39:17 +00:00
Eric Traut 08b07eef12 Preserve daemon diagnostics and include updater logs in reports (#49843)
## Why

Detached launches truncate stderr logs, losing details from previous failures. Updater handoffs can also leave existing writers with stale offsets after truncation.

## What changed

- Record daemon lifecycle and update stages with timings, outcomes, and typed OS errors, omitting error messages and context.
- Preserve the last 256 KiB of nonempty stderr logs before truncation and use append handles to prevent gaps during handoff. Preservation failures do not block launch.
- Attach current and previous updater log tails when log inclusion is enabled for a report, with a 512 KiB total limit and legacy filename fallback. Skip missing, empty, and nonregular files.

## Testing

Add regression coverage for diagnostic error redaction, bounded log preservation, append behavior during handoff, attachment selection, and updater logs in captured report uploads.

GitOrigin-RevId: 918317022ab9dcfd1b13d676872706c77e9ffbb2
2026-10-01 03:24:18 +00:00
Eric Traut cda82a2c68 Recover daemon startup and updater re-exec after cwd deletion (#49819)
## Why

Managed daemons and updaters can outlive their launching directory. Restarts and updater replacement need to keep working after that directory is removed.

## What changed

- Share working-directory preparation between background launches and Unix updater re-execution. Fall back to the daemon state directory when the Unix working directory is missing, while retaining usable working directories and the existing Windows state-directory behavior.
- Preserve configured `CODEX_HOME` paths and valid symlink aliases, and make certificate and AWS environment paths absolute before launching children. Handle both `npm_config_cafile` and `NPM_CONFIG_CAFILE`, preserving supported home-relative values.
- Log warnings when scheduled daemon updates fail.

## Testing

Add Unix regression coverage for restarting the daemon and updater from a deleted working directory, and for updater re-execution with preserved environment paths, `CODEX_HOME` symlink aliases, and a Linux non-Unicode home path.

GitOrigin-RevId: f45aefd8ff060adc6e025badf0bb8c81f842bf88
2026-10-01 02:06:50 +00:00
Eric Traut 321dcf5a6f Allow daemon updates to restore pinned packages to latest stable (#45780)
## Why

`codex app-server daemon update` previously required a latest-channel installation, leaving pinned and local managed packages unable to return to production updates through that command.

## What changed

- Allow explicit updates to restore managed packages to the latest stable release while preserving the automatic-update preference. Extend legacy migration to local and pinned packages when the published installer and release support it.
- Bind restoration to the selected release with a single-use updater authorization and installer guards. Scheduled updates and ordinary updater socket requests continue to respect pins.
- Restart a running daemon when its selected package changes, even if the binary and version are identical; leave stopped daemons stopped.
- Reject incompatible production packages before selecting them, resolve Windows junction targets for ownership checks, and show the restoration command after installing a pinned CLI package.

## Testing

Extend daemon and installer tests to cover local-package migration, restoration with automatic updates enabled or disabled, same-binary package restarts, stopped-daemon preservation, selection races, incompatible releases, and socket requests that must not undo pins.

GitOrigin-RevId: d64520c1190b5c7bac084fb4a1b6b24998b08dfa
2026-09-15 19:51:06 +00:00
Eric Traut c0316291ca Use native process identities for PID-managed daemons (#45779)
## Why

Locale, timezone, or system clock changes can alter the `ps` start-time text used to identify a running daemon, causing its PID record to be treated as stale.

## What changed

- Record and check native process identities on Linux and macOS, using boot IDs to reject records from previous boots and native process details to detect PID reuse.
- Keep `processStartTime` for older clients and promote verifiable legacy daemon and updater records when the updater starts.
- Retain legacy records and report an error when a live process's start-time text no longer matches. Propagate Windows process-access errors instead of treating inaccessible PIDs as stale.

## Testing

Add regression coverage for locale and timezone changes, altered legacy timestamps, legacy record promotion, PID reuse, previous boots, and macOS process ownership differences. Extend zombie-reaping coverage to native identities and verify that Windows access-denied errors are preserved.

GitOrigin-RevId: b5752497c0e8186bb604fe5dd33f570acc068f20
2026-09-15 19:46:37 +00:00
Eric Traut 923c6028b6 Move daemon packages out of the standalone CLI installation (#45546)
## Why

The daemon shares the standalone CLI package, coupling daemon updates to the visible CLI installation. A dedicated package lets daemon updates preserve the CLI selection and shell profile.

## What changed

- Add dedicated daemon package selection and separate daemon and updater PID files, with diagnostics for the new files.
- Make `codex app-server daemon update` migrate eligible legacy stable latest-channel installations to a dedicated package. Prepare and validate the release and its package-ownership support before switching; preserve settings and whether the daemon was running. Scheduled updates keep legacy installations in place.
- Add daemon-only and deferred-selection modes to the shell and PowerShell installers, leaving the visible CLI installation unchanged.
- Coordinate migration with installer locks, refresh package selection after acquiring lifecycle locks, and reject package changes during update downloads.

## Testing

Add coverage for running and stopped migrations, incompatible installers and releases, package-selection races, daemon-only updates, and a capability probe that does not start an updater. Add a shell installer regression test verifying that daemon installation preserves the visible CLI and shell profile.

GitOrigin-RevId: 1ec0f358f8f6ebde7daddb74e880cc8d2235d647
2026-09-14 23:43:59 +00:00
Eric Traut daca1fab84 Add an explicit app-server daemon update command (#43562)
## What changed

Add `codex app-server daemon update` to check the latest stable release once, even when automatic updates are disabled. The command requires an installer-owned latest-channel standalone installation and reports `updated`, `noUpdate`, or `unsupported` as JSON, including installed and running versions.

Route manual requests through the updater so scheduled and manual installs share one owner and accepted updates continue if the CLI exits. Restart a running managed daemon when its binary or version differs from the selected installation; this may interrupt active or queued work. Return installer failures as command errors and retry requests across updater replacement.

## Testing

Add coverage for manual updates with automatic updates disabled, queued requests, same-version binary replacement, updater handoff retries, and unsupported installations or unmanaged servers.

GitOrigin-RevId: 240b63c79bae3a64a80768131a30da70900a4d62
2026-09-07 19:11:41 +00:00
Eric Traut 769a6a5bcd Record the launched app-server executable identity in PID files (#43552)
## Why

An installer can retarget the selected executable symlink or junction while an app server is running. Recording the launch-time identity preserves which binary was started after that path changes.

## What changed

- Resolve the executable path before launching on all platforms and store an optional executable identity in app-server PID records. Continue accepting older records without this field and expose the identity only for an active process.
- Use BLAKE3 for executable identity digests and make `ExecutableIdentity` serializable.
- Schedule updater checks with an explicit deadline in `tokio::select!`, preserving persisted update intervals, settings-error retries, and termination handling.

## Testing

Add a Unix test that verifies the recorded identity matches the resolved launch binary, differs from a retargeted symlink's binary, and remains optional when reading legacy PID records.

GitOrigin-RevId: 4821130b9a5e75b417a8e39c583b30427a872ff8
2026-09-07 18:52:37 +00:00
Eric Traut 7d8e2dd6c5 Make app-server daemon automatic updates configurable (#43542)
## Why

The managed daemon's updater used a fixed hourly cadence with no saved preference to disable automatic updates.

## What changed

- Read `updater.autoUpdateEnabled` and `updater.updateIntervalMinutes` from `CODEX_HOME/app-server-daemon/settings.json`, defaulting to enabled with a 60-minute interval. Keep the initial five-minute delay and require a positive interval.
- Apply the enabled preference during daemon lifecycle operations and re-read settings before updates and recurring waits. Preserve updater preferences and unknown fields when saving remote-control settings, using atomic file replacement.
- Signal the updater's process group on Unix so stopping it also terminates installer children. Allow daemon shutdown even when settings are malformed.
- Document updater configuration and remote-control preference behavior. Manual `codex update` remains unaffected.

## Testing

Add tests for settings preservation, interval validation, recurring waits, enabling and disabling updates across restart and bootstrap, Unix installer termination, and shutdown with malformed settings.

GitOrigin-RevId: 5dbb72c5939e9ced2390b3760a7265608c8b61a0
2026-09-07 18:26:04 +00:00
Eric Traut adee0b04fa Preserve standalone release pins during daemon updates (#43521)
## Why

Bootstrapping the app-server daemon should preserve an explicitly selected release, even when that version is currently `latest`. Older managed binaries should also be able to serve app-server without supporting the updater command.

## What changed

- Record `latest` selections in `auto-update-version` in both standalone installers and clear the marker for explicit releases.
- Start the daemon updater only for a marked stable release whose binary supports `pid-update-loop`. Existing installs without a marker require a new `latest` installation to enable automatic updates.
- Recheck the selected release under the install lock so an in-flight update cannot overwrite a new pin, including installer calls from older updaters. Recheck selection before restarting app-server or replacing the updater.
- Cancel Unix installer process groups and clean up their owned fallback locks when the updater stops.

## Testing

Add coverage for channel markers, explicit pins of the current latest version, older updater guards, updater command support, and Unix installer cancellation with child-process and lock cleanup.

GitOrigin-RevId: 4d275237bd77d896bf64dc1b85a5bca3608142c7
2026-09-07 17:18:02 +00:00
Eric Traut 91608236ea Support managed daemon updates on Windows (#42392)
## What changed

- Run the daemon update loop on Windows using the non-interactive PowerShell installer.
- Restart the managed app server after updates and hand updater ownership to a changed managed executable through a readiness handshake.
- Contain installer descendants in a Windows job while allowing app-server and successor updater processes to break away.
- Preserve or restore PID ownership when successor startup fails, without terminating a process that reused the PID.

## Testing

- Cover non-interactive installer success and script failure on Windows.
- Cover updater handoff, readiness, rollback, inaccessible processes, and PID reuse.

GitOrigin-RevId: 5178c01124b4dd86b507b53f5093ec9a9d48e12c
2026-09-03 00:07:34 +00:00
Michael Bolin 99eb575649 Honor system proxy settings in the daemon updater (#34495)
## Why

The PID-managed daemon updater downloaded the standalone installer directly,
bypassing Codex's configured outbound proxy policy.

## What changed

- Build the updater's HTTP client from the effective Codex configuration so
  `features.respect_system_proxy` also applies to installer downloads.
- Use the route-aware client pool for the installer request.
- Warn and fall back to the default `reqwest` proxy behavior when configuration
  cannot be loaded, allowing updates to continue.

## Testing

Added tests for proxy-policy selection, configuration-load fallback, exact
installer URL usage, byte preservation, and non-success HTTP responses.

GitOrigin-RevId: 559668eb40314d43afa6512da66e59e32cc1b93c
2026-07-21 08:50:21 +00:00
Ruslan Nigmatullin e3f481da98 daemon: refresh updater after validated binary rollout (#21853)
## Why

`bootstrap` starts a detached pid-backed updater loop, but before this
change that updater could keep running an old executable image even
after `install.sh` replaced the managed standalone binary under
`CODEX_HOME`. That left the updater itself behind the binary it had just
rolled out, especially when the app-server was stopped or when the
managed binary changed without a version-string change.

## What changed

- Track updater identity from the executable contents rather than only
the reported CLI version.
- Force the managed app-server restart path when the managed binary
contents differ from the running updater image, then re-exec the updater
from the managed binary once the rollout is in a safe state.
- Distinguish a genuinely absent managed app-server from a managed
process that exists but is not yet probeable, so self-refresh does not
skip a required restart.
- Keep the restart/re-exec decision under the daemon operation lock so
`bootstrap` cannot race the handoff.
- Update `app-server-daemon/README.md` to document the resulting
standalone and out-of-band update behavior.

## Verification

- `cargo test -p codex-app-server-daemon`
- `just fix -p codex-app-server-daemon`

Added focused unit coverage for:
- content-based updater refresh decisions
- safe updater re-exec outcomes across restart states
2026-05-11 12:37:10 -07:00
Ruslan Nigmatullin 0c8d42525e [daemon] Add app-server daemon lifecycle management (#20718)
## Why

Desktop and mobile Codex clients need a machine-readable way to
bootstrap and manage `codex app-server` on remote machines reached over
SSH. The same flow is also useful for bringing up app-server with
`remote_control` enabled on a fresh developer machine and keeping that
managed install current without requiring a human session.

## What changed

- add the new experimental `codex-app-server-daemon` crate and wire it
into `codex app-server daemon` lifecycle commands: `start`, `restart`,
`stop`, `version`, and `bootstrap`
- add explicit `enable-remote-control` and `disable-remote-control`
commands that persist the launch setting and restart a running managed
daemon so the change takes effect immediately
- emit JSON success responses for daemon commands so remote callers can
consume them directly
- support a Unix-only pidfile-backed detached backend for lifecycle
management
- assume the standalone `install.sh` layout for daemon-managed binaries
and always launch `CODEX_HOME/packages/standalone/current/codex`
- add bootstrap support for the standalone managed install plus a
detached hourly updater loop
- harden lifecycle management around concurrent operations, pidfile
ownership, stale state cleanup, updater ownership, managed-binary
preflight, Unix-only rejection, forced shutdown after the graceful
window, and updater process-group tracking/cleanup
- document the experimental Unix-only support boundary plus the
standalone bootstrap/update flow in
`codex-rs/app-server-daemon/README.md`

## Verification

- `cargo test -p codex-app-server-daemon -p codex-cli`
- live pid validation on `cb4`: `bootstrap --remote-control`, `restart`,
`version`, `stop`

## Follow-up

- Add updater self-refresh so the long-lived `pid-update-loop` can
replace its own executable image after installing a newer managed Codex
binary.
2026-05-08 16:51:16 -07:00