Enable configurable chunked tus uploads and related robustness changes.
- Add AGENT_TUS_CHUNK_SIZE_BYTES env (default 1 MiB, 0 disables chunking) and docs in machinery/.env; ignore machinery/go.work files and set GOWORK=off in VSCode launch to avoid go.work during debugging.
- Implement tusChunkSize() and update uploadVaultResumable to send PATCHes in configurable chunk sizes, checkpoint progress after each chunk, and handle partial failures by refreshing retry budget when progress occurs.
- Change tusPatch to accept an explicit length and return the advanced offset when a PATCH is fully accepted.
- Skip uploads when the recording file no longer exists to avoid infinite retries.
- Add tests exercising chunked uploads, chunking-disabled behavior, and tusChunkSize parsing; extend fakeTus test server to record patch sizes.
- Reduce noisy info logs to debug in AAC transcoder and WebRTC audio processing.
These changes improve resumable upload reliability, allow tuning for proxy/load-balancer limits, and reduce log spam during normal operation.
Introduce a standalone mp4analyze CLI for inspecting fragmented MP4s and add a full-featured tus resumable upload client used by Kerberos Vault uploads.
Changes:
- Add machinery/cmd/mp4analyze/main.go: CLI tool to analyze MP4 structure, fragments, keyframes, NALs and SPS/PPS differences.
- Add machinery/src/cloud/tus_client.go: implements tus 1.0.0 client with create/head/patch/delete, sidecar resume state, metadata encoding, Location resolution, backoff, and helpers (newVaultHTTPClient, setVaultTusHeaders). Resumable uploads are enabled by default and can be disabled via AGENT_DISABLE_RESUMABLE_UPLOAD. Sidecar files are stored under data/tus.
- Add machinery/src/cloud/tus_client_test.go: in-memory fake tus server and unit tests exercising happy path, unsupported servers, finalize-retry and resume-from-sidecar behavior, and helpers.
- Refactor machinery/src/cloud/kerberos_vault.go: replace inlined POST upload logic with sendToVault which attempts resumable uploads first and falls back to legacy single-POST (uploadVaultLegacy). Improve retry semantics so retry counters only advance on definitive vault responses, centralize header setup, and use new HTTP client builder honoring AGENT_TLS_INSECURE.
The change preserves backward compatibility with vaults that do not support tus by transparently falling back to the legacy upload path. Tests cover core tus behaviors and resume semantics.
Buffer and conditionally drop a pending GOP when an upstream loop/restart emits a premature IDR. mp4.go: add gopBuffer and bufferedSample types, hold samples until the next video keyframe, detect a seam by comparing the new keyframe interval against the previous cadence (SeamGapDivisor) and drop the short/truncated tail GOP or commit buffered samples. Add commitBufferedGOP and commitSampleToTrack helpers and flush the final buffered GOP on Close. mp4_loopseam_test.go: update test descriptions, expectations and names to assert the truncated tail GOP is dropped exactly once across different GOP sizes. Dockerfile and Dockerfile.arm64: re-declare ARG VERSION inside the build stage and only derive git describe when VERSION is unset or the default 0.0.0 so build-arg values are respected. Add a binary MP4 fixture used by the tests.
Add logic to detect an upstream source loop/restart seam (premature IDR) and force a fragment flush so the seam IDR starts its own fragment. Introduces SeamGapDivisor constant and two MP4 fields (LastKeyframeRawPTS, LastKeyframeGapMs) to track recent keyframe timing; when a keyframe interval is significantly shorter than the previous interval (gap*SeamGapDivisor < previousGap) a fragment boundary is forced and a warning is logged. This approach derives the threshold from the observed GOP cadence to avoid false positives on short-GOP or all-intra streams.
Also add tests (machinery/src/video/mp4_loopseam_test.go) that synthesize loop-seam scenarios across multiple GOP sizes (15, 30, 60 frames) to verify the seam is isolated, and include a sample MP4 reproducer (machinery/thales_1781183923_3-758_2top_0-0-0-0_-1_30221.mp4).
When prepending H.264 parameter sets to keyframes, build a fresh buffer instead of appending into existing slices to avoid corrupting shared backing arrays (which could produce an invalid avcC and trigger FFmpeg "non-existing PPS 0 referenced" errors). Also add explicit error logs in mp4.Close() to surface incomplete H.264/H.265 parameter sets (avcC/hvcC) so missing VPS/SPS/PPS conditions are easier to diagnose.
Replace 'secrets: inherit' with an explicit secrets mapping in the PR description workflow. This exposes TOKEN and the Azure/OpenAI-related secrets (AZURE_OPENAI_API_KEY, OPENAI_MODEL, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_VERSION) to the job so it can authenticate against the OpenAI/Azure services when generating or updating PR descriptions.
Replace the inline OpenAI-based PR description job with a reusable workflow call (uug-ai/workflows/.github/workflows/pr-description.yml@main). Removed unused env vars and the checkout/action steps, and pass pr_number and overwrite_description inputs while inheriting repository secrets. This centralizes PR description logic and simplifies the workflow file.
Introduce streamHealth instrumentation for Golibrtsp to track per-stream metrics (write durations, gaps, lost packets, decode errors) and emit immediate warnings and periodic summaries to distinguish downstream back-pressure from upstream camera/network stalls. Wire gortsplib Client hooks (OnPacketsLost, OnDecodeError) to the health logger, label streams, and measure WritePacket blocking time around packet writes. Add packetAgeString helper and augment Kerberos ControlAgent logs with stalled counters and last-packet age to provide more context when triggering restarts.
Compute and propagate per-sample composition time offsets (PTS - DTS) so fragmented MP4s remain decode-timestamp-monotonic while preserving presentation order for B-frame streams. Changes include:
- Add a dtsExtractor interface and compositionOffsetMs helper to extract DTS from H264/H265 access units using mediacommon extractors (safe no-op if extraction fails).
- Compute composition offsets for H264 and H265 packet handlers and store them in Packet.CompositionTime. Preserve decoded AU for H265 before Annex-B rewriting for correct DTS extraction.
- Introduce writeSampleToMP4 helper to centralize writing logic; derive DTS = PTS - compositionOffset when present and call MP4.AddSampleToTrack accordingly.
- Change MP4.AddSampleToTrack signature to accept compositionOffset and write it into sample.CompositionTimeOffset so players (MSE) can present samples in PTS order while fragments use DTS.
- Update tests to pass the new compositionOffset argument.
This ensures proper playback of streams with frame reordering (B-frames) in browsers and other fragmented-MP4 consumers.
When the agent configuration is provided via environment variables (e.g. Kubernetes ConfigMap in factory standalone/configmap mode) there is no separate custom config from MongoDB. Add logic in OverrideWithEnvironmentVariables to copy configuration.Config into configuration.CustomConfig when DEPLOYMENT is unset or set to "agent", so UI/consumers that read the per-agent custom configuration (such as the factory agent edit page) see the env-injected values instead of an empty config.
UploadKerberosHub used a bare http.Client with no CheckRedirect policy, so
it followed redirects automatically. net/http strips the standard sensitive
headers on a cross-host redirect but not custom-named headers, so the Hub
credentials carried in X-Kerberos-Hub-PrivateKey / X-Kerberos-Hub-PublicKey
were forwarded verbatim to any host the configured HubURI redirected to,
disclosing the private key.
Add a CheckRedirect policy that deletes the Hub credential headers when the
redirect target host differs from the original request host.
Signed-off-by: tonghuaroot <tonghuaroot@gmail.com>
