Commit Graph
10 Commits
Author SHA1 Message Date
jif 4be9cfdf30 Improve diagnostics for compressed rollout read failures (#55545)
## Why

Undecodable compressed rollouts need diagnostic details and their original bytes preserved for investigation.

## What changed

- Log compressed file size, first-frame content size, decoded byte count, and whether the open file changed when a compressed rollout read fails. Limit warnings to once per process per minute and exclude paths and contents.
- Allow diagnostic reports to attach the raw `.jsonl.zst` archive after a decoding failure when no preferred plain rollout exists. Retain attachment size limits and append `.zst` to filename overrides when needed.

## Testing

Add coverage for truncated streams, incomplete trailing frame headers, files truncated after opening, and diagnostic uploads that preserve undecodable archives with size limits and filename overrides.

GitOrigin-RevId: 76f7ee466f9bebdab5a91ce1894f4361749edc5b
2026-10-20 11:42:48 +00:00
jif 5fe32c0001 Preserve rollout access metadata across compression and restoration (#55252)
## Why

Replacing rollout files can lose Unix ownership or macOS ACLs, changing who can read or append to a transcript. An ACL can also permit appending while denying explicit timestamp updates.

## What changed

- Preserve Unix ownership, permissions, modification times, and macOS ACLs before publishing compressed or restored rollouts, and verify the replacement can be reopened.
- Abort replacement on metadata failures, reject macOS delete-deny ACLs to keep temporary files removable, and detect Unix access metadata changes through `ctime` checks.
- Use managed restoration temporary files whose names avoid stale `*.tmp` cleanup.
- Allow appending when refreshing the rollout's modification time returns `PermissionDenied`.

## Testing

Add regression coverage for macOS ACL preservation, inherited ACL removal, deletion-protected rollouts, append access despite denied timestamp changes, Unix group preservation, and restoration temporary files surviving stale-file cleanup.

GitOrigin-RevId: 23f3de6e35c0b2e19f3491170e320159ea3bdb6f
2026-10-19 11:02:40 +00:00
Charlie Marsh a9bc7bebfa Load rollout history on a single blocking worker (#51499)
## What changed

Move full rollout history reading and parsing into one blocking worker for both plain `.jsonl` and compressed `.jsonl.zst` files. Use the shared cancellation-aware line iterator so canceled loads stop before processing another record, while preserving rollout items, thread IDs, and parse-error counts.

## Testing

Add regression tests for cancellation between records and equivalent history-loading results across plain and compressed files, including Unicode, CRLF line endings, malformed records, empty files, and invalid UTF-8.

GitOrigin-RevId: 65e1bd7ebcdbbd0ab5f560e73b2c7f074f627f8f
2026-10-06 23:04:02 +00:00
Charlie Marsh 5aa92804d2 Batch rollout listing scans on cancellable blocking workers (#49694)
## What changed

Move directory enumeration and candidate metadata lookups into blocking workers, batching updated-time candidate collection into one worker per scan. Check cancellation between entries and metadata lookups so dropping the async waiter stops further scanning.

Share synchronous rollout metadata resolution with the async helper, preserving plain-file precedence over compressed siblings. Retain scan limits, stable file ordering, and directory-level early stopping for creation-time listings.

## Testing

Add tests for cancellation during directory enumeration and candidate discovery in both flat and date-partitioned layouts, including plain-sibling metadata precedence and filtering of unrelated files, directories, and symlinks.

GitOrigin-RevId: 324289003b29a7d2b3220da03e3c002ab5e23eec
2026-09-30 18:03:48 +00:00
npancha-openai d1c4e3c3e6 Keep compressed rollout snippet searches on one blocking worker (#49692)
## Why

Compressed snippet searches schedule a blocking task for every line read. Keeping the scan on one worker avoids repeated task scheduling.

## What changed

Use `RolloutLineReader::find_map` to scan compressed rollouts on a single blocking worker, stopping at the first matching snippet. Preserve read metrics and stop delivering records when the search is canceled, including cancellation during a read.

## Testing

Add tests for first-match snippets, case-insensitive literal matching, malformed records, missing matches, stream errors before a match, and cancellation between lines or during a read.

GitOrigin-RevId: 295206b1a00e861bc44606bd843f6ed8c3e6aae1
2026-09-30 17:57:58 +00:00
jif d6c4c6aea4 Classify rollout read failures by reason and progress (#47565)
## Why

Rollout read metrics record the failure stage and I/O error kind, but these do not distinguish specific zstd failures, task join failures, or a busy reader, nor indicate whether any line was read before failure.

## What changed

Add bounded `reason` and `read_progress` labels to rollout read counters and duration metrics, captured at the first failure. Distinguish OS errors, reader-busy and task-join failures, and known zstd error categories, with `stream_error` as the fallback.

Record whether the failure occurred before or after the first line. Classify blocking-task join failures during both open and read. Keep error messages, file paths, and rollout contents out of metric labels.

GitOrigin-RevId: 6db302900f504bc2860504d81c18017adcef3efc
2026-09-23 15:01:19 +00:00
jif 821ad43f9d Tag rollout compression metrics by trigger (#46047)
## Why

Compression metrics do not distinguish startup runs from runs requested through
`rollout/compress`, making their outcomes indistinguishable by entry point.

## What changed

Pass `RolloutCompressionTrigger` from startup and RPC callers through the
compression worker. Add a `trigger` tag with `startup` or `rpc` to run, file,
scan-error, and temporary-file cleanup metrics, including durations, byte counts,
and compression ratios.

GitOrigin-RevId: f23d9da791b666a7311f2323f6bead6570067798
2026-09-16 22:33:15 +00:00
jif da18000cae Measure rollout read and materialization durations (#45966)
## What changed

- Emit one read counter and duration observation per rollout reader, tagged by format, outcome, failure stage, and error kind. Distinguish EOF, partial reads, and failures, including failed opens.
- Accumulate time spent in completed open/retry and read calls, excluding caller processing and canceled calls, without exporting a metric for each JSONL record.
- Record materialization duration when decompressing a rollout for append, for both successful decompression and failures.

GitOrigin-RevId: 3d1e257863ffa696a63040172df54d8234d56ed5
2026-09-16 15:36:43 +00:00
jif 2d90e054d6 Expose partial completion in rollout compression metrics (#45964)
## Why

A compression run can report `completed` after hitting its time budget or encountering file, scan, or cleanup errors. The run metrics do not distinguish these cases from a clean, complete pass.

## What changed

- Add `completion_reason` (`time_budget` or `scan_finished`) and `file_errors`, `scan_errors`, and `cleanup_errors` tags to completed run counts and durations, preserving `status=completed`.
- Record scan failures and additional stale-temp cleanup failures with operation stages and bounded I/O error kinds, and track recoverable errors in the run summary.

GitOrigin-RevId: bb6c33890371a2cdb02bc6ccc82a5df343b823b5
2026-09-16 15:26:30 +00:00
jif 3fa9039bd7 Label rollout compression failures by stage and I/O error kind (#45461)
## Why

Rollout compression failure counters report only that an operation failed,
without identifying the failing stage or I/O error kind.

## What changed

- Add `stage` and `error_kind` labels to failure counters for compression runs,
  individual files, materialization for append, and stale temporary file cleanup.
- Record failures at their source, including lock acquisition and task joins,
  and avoid counting file compression failures twice.
- Use static stage labels and a fixed set of error categories, keeping error
  messages, paths, and rollout contents out of metric tags.

GitOrigin-RevId: ac2bfc7ae4cec4e9f60de9557b03345a3986aa53
2026-09-14 16:37:47 +00:00