mirror of
https://github.com/kerberos-io/agent.git
synced 2026-11-21 22:50:39 +00:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
57055498a2 |
@@ -1,5 +1,9 @@
|
||||
FROM mcr.microsoft.com/devcontainers/go:1.26-trixie
|
||||
|
||||
# Same pin as the main Dockerfile: Go 1.26.11 rejects moq-go-ffi v0.3.x's
|
||||
# archives (LLVM bitcode), which breaks `go build -tags moq`.
|
||||
ENV GOTOOLCHAIN=go1.26.10
|
||||
|
||||
# Install node environment
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
|
||||
+4
-1
@@ -1,5 +1,8 @@
|
||||
|
||||
ARG GO_IMAGE=golang:1.26-trixie
|
||||
# Pinned to 1.26.10: Go 1.26.11 rejects C archives containing LLVM bitcode
|
||||
# ("disallowed LTO section .llvmbc"), and moq-go-ffi v0.3.x ships such archives.
|
||||
# Unpin once the MoQ code moves to moq.dev/moq (moq-ffi v0.4.27+ is clean).
|
||||
ARG GO_IMAGE=golang:1.26.10-trixie
|
||||
ARG RUNTIME_IMAGE=debian:trixie-slim
|
||||
ARG VERSION=0.0.0
|
||||
FROM ${GO_IMAGE} AS build-machinery
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# Recording file structure
|
||||
|
||||
This document describes where the Kerberos Agent stores recordings, locally and in
|
||||
direct S3 uploads, and how the current layout relates to the legacy one.
|
||||
|
||||
## Local recording layout
|
||||
|
||||
New local recordings are stored under
|
||||
`data/recordings/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4`, using the recording's UTC
|
||||
start date and Unix-millisecond timestamp. For example:
|
||||
`data/recordings/2026/01/02/1767355200123-device-key.mp4`.
|
||||
The filename no longer embeds the camera name, region, change count, or duration,
|
||||
and it does not change when the recording is finalized.
|
||||
Those attributes, together with FPS and encryption status, are captured in the
|
||||
`.metadata` upload checkpoint and sent as structured HTTP/tus metadata.
|
||||
|
||||
Existing flat recordings and dated recordings with older long filenames remain
|
||||
supported without a migration. Recording lists, day filters, playback, uploaders,
|
||||
and automatic cleanup handle both filename formats and layouts. Upload queue
|
||||
checkpoints remain flat in `data/cloud/`. Successful-upload removal also resolves
|
||||
both local layouts. Deploy the updated Vault alongside Agent for canonical
|
||||
`<account-directory>/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4` storage keys and
|
||||
metadata-driven Hub integration events.
|
||||
Directory decryption also processes dated subdirectories and writes each output
|
||||
to a sibling `decrypted/` directory; those outputs are excluded from recording
|
||||
lists and automatic cleanup.
|
||||
|
||||
## Direct S3 recording layout
|
||||
|
||||
Direct S3 uploads store recordings as
|
||||
`<s3.username>/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4`. The timestamp is the
|
||||
recording's Unix timestamp in milliseconds, and the date is its UTC date, not the upload
|
||||
date. Timestamp and device key come from the recording's `.metadata` upload
|
||||
checkpoint, which stores the timestamp in milliseconds. The uploader also
|
||||
preserves checkpoint duration, FPS (when available), and encryption status as S3
|
||||
object metadata. The device key must be non-empty and contain no path separators.
|
||||
Retried uploads use the same object key, even after midnight. Millisecond
|
||||
precision keeps recordings starting within the same second distinct.
|
||||
|
||||
This replaces the previous `<s3.username>/<filename>.mp4` layout for new uploads.
|
||||
Existing objects are not moved; consumers of direct S3 uploads must support both
|
||||
layouts when reading historical recordings. Updated Kerberos Vault deployments
|
||||
use the same filename and dated layout under the authenticated account directory
|
||||
for both direct Vault uploads and Hub-proxied uploads.
|
||||
|
||||
Only historical recordings with missing, empty, or FPS-only queue markers fall
|
||||
back to the legacy filename format and configured Agent key. A structured
|
||||
checkpoint takes precedence over both the filename and current configured key;
|
||||
an invalid or unreadable checkpoint is reported as an upload error, not silently
|
||||
replaced with filename-derived values.
|
||||
For older structured checkpoints, missing camera name, region, and change count
|
||||
are recovered from a validated legacy filename. Populated checkpoint fields
|
||||
take precedence, and the checkpoint timestamp and device key remain authoritative.
|
||||
|
||||
The shorter filename cannot be decoded by legacy filename-only consumers.
|
||||
Integrations must preserve structured recording metadata, including the device
|
||||
key (`productid`) and recording timestamp (`event-timestamp`), as well as the full
|
||||
object key.
|
||||
@@ -61,6 +61,7 @@ There are a myriad of cameras out there (USB, IP and other cameras), and it migh
|
||||
|
||||
1. [Encryption](#encryption)
|
||||
2. [H264 vs H265](#h264-vs-h265)
|
||||
3. [Recording file structure](README-FILE-STRUCTURE.md)
|
||||
|
||||
### Contributing
|
||||
|
||||
@@ -76,60 +77,6 @@ There are a myriad of cameras out there (USB, IP and other cameras), and it migh
|
||||
1. [What is new?](#what-is-new)
|
||||
1. [Contributors](#contributors)
|
||||
|
||||
### Direct S3 recording layout
|
||||
|
||||
Direct S3 uploads store recordings as
|
||||
`<s3.username>/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4`. The timestamp is the
|
||||
recording's Unix timestamp in milliseconds, and the date is its UTC date, not the upload
|
||||
date. Timestamp and device key come from the recording's `.metadata` upload
|
||||
checkpoint, which stores the timestamp in milliseconds. The uploader also
|
||||
preserves checkpoint duration, FPS (when available), and encryption status as S3
|
||||
object metadata. The device key must be non-empty and contain no path separators.
|
||||
Retried uploads use the same object key, even after midnight. Millisecond
|
||||
precision keeps recordings starting within the same second distinct.
|
||||
|
||||
This replaces the previous `<s3.username>/<filename>.mp4` layout for new uploads.
|
||||
Existing objects are not moved; consumers of direct S3 uploads must support both
|
||||
layouts when reading historical recordings. Updated Kerberos Vault deployments
|
||||
use the same filename and dated layout under the authenticated account directory
|
||||
for both direct Vault uploads and Hub-proxied uploads.
|
||||
|
||||
Only historical recordings with missing, empty, or FPS-only queue markers fall
|
||||
back to the legacy filename format and configured Agent key. A structured
|
||||
checkpoint takes precedence over both the filename and current configured key;
|
||||
an invalid or unreadable checkpoint is reported as an upload error, not silently
|
||||
replaced with filename-derived values.
|
||||
For older structured checkpoints, missing camera name, region, and change count
|
||||
are recovered from a validated legacy filename. Populated checkpoint fields
|
||||
take precedence, and the checkpoint timestamp and device key remain authoritative.
|
||||
|
||||
The shorter filename cannot be decoded by legacy filename-only consumers.
|
||||
Integrations must preserve structured recording metadata, including the device
|
||||
key (`productid`) and recording timestamp (`event-timestamp`), as well as the full
|
||||
object key.
|
||||
|
||||
### Local recording layout
|
||||
|
||||
New local recordings are stored under
|
||||
`data/recordings/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4`, using the recording's UTC
|
||||
start date and Unix-millisecond timestamp. For example:
|
||||
`data/recordings/2026/01/02/1767355200123-device-key.mp4`.
|
||||
The filename no longer embeds the camera name, region, change count, or duration,
|
||||
and it does not change when the recording is finalized.
|
||||
Those attributes, together with FPS and encryption status, are captured in the
|
||||
`.metadata` upload checkpoint and sent as structured HTTP/tus metadata.
|
||||
|
||||
Existing flat recordings and dated recordings with older long filenames remain
|
||||
supported without a migration. Recording lists, day filters, playback, uploaders,
|
||||
and automatic cleanup handle both filename formats and layouts. Upload queue
|
||||
checkpoints remain flat in `data/cloud/`. Successful-upload removal also resolves
|
||||
both local layouts. Deploy the updated Vault alongside Agent for canonical
|
||||
`<account-directory>/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4` storage keys and
|
||||
metadata-driven Hub integration events.
|
||||
Directory decryption also processes dated subdirectories and writes each output
|
||||
to a sibling `decrypted/` directory; those outputs are excluded from recording
|
||||
lists and automatic cleanup.
|
||||
|
||||
## Quickstart - Docker
|
||||
|
||||
The easiest way to get your Kerberos Agent up and running is to use our public image on [Docker hub](https://hub.docker.com/r/kerberos/agent). Once you have selected a specific tag, run `docker` command below, which will open the web interface of your Kerberos agent on port `80`, and off you go. For a more configurable and persistent deployment have a look at [Running and automating a Kerberos Agent](#running-and-automating-a-kerberos-agent).
|
||||
|
||||
+2
-2
@@ -27,6 +27,7 @@ require (
|
||||
github.com/kerberos-io/joy4 v1.0.64
|
||||
github.com/kerberos-io/onvif v1.2.3
|
||||
github.com/minio/minio-go/v6 v6.0.57
|
||||
github.com/moq-dev/moq-go v0.5.7
|
||||
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
|
||||
github.com/pion/interceptor v0.1.47
|
||||
github.com/pion/rtcp v1.2.17
|
||||
@@ -46,7 +47,6 @@ require (
|
||||
go.opentelemetry.io/otel/sdk v1.47.0
|
||||
go.opentelemetry.io/otel/trace v1.47.0
|
||||
golang.org/x/net v0.62.0
|
||||
moq.dev/moq v0.7.27
|
||||
)
|
||||
|
||||
require (
|
||||
@@ -99,6 +99,7 @@ require (
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
||||
github.com/modern-go/reflect2 v1.0.2 // indirect
|
||||
github.com/montanaflynn/stats v0.7.1 // indirect
|
||||
github.com/moq-dev/moq-go-ffi v0.3.7 // indirect
|
||||
github.com/nxadm/tail v1.4.11 // indirect
|
||||
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
|
||||
github.com/pion/datachannel v1.6.2 // indirect
|
||||
@@ -142,5 +143,4 @@ require (
|
||||
gopkg.in/ini.v1 v1.42.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
howett.net/plist v0.0.0-20181124034731-591f970eefbb // indirect
|
||||
moq.dev/moq-ffi v0.4.27 // indirect
|
||||
)
|
||||
|
||||
+4
-4
@@ -857,6 +857,10 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
|
||||
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
|
||||
github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE=
|
||||
github.com/montanaflynn/stats v0.7.1/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
|
||||
github.com/moq-dev/moq-go v0.5.7 h1:LfFpgAU8FRMcnU85L5Lb03HzxhYbQ2+BhT/pJQdjy5U=
|
||||
github.com/moq-dev/moq-go v0.5.7/go.mod h1:5K8zjKKjWe5lzfCtlfxoAfdCr3KX6EBuTOZsz0WBnzw=
|
||||
github.com/moq-dev/moq-go-ffi v0.3.7 h1:+xwPOzTJHvB0tuTnW6znbvvOgT4yLnN7sMqXXZYh02M=
|
||||
github.com/moq-dev/moq-go-ffi v0.3.7/go.mod h1:zxpOlUetvaoxWBnbXTdILtLLUytaoVuLvO36lftwWO0=
|
||||
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 h1:zYyBkD/k9seD2A7fsi6Oo2LfFZAehjjQMERAvZLEDnQ=
|
||||
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646/go.mod h1:jpp1/29i3P1S/RLdc7JQKbRpFeM1dOBd8T9ki5s+AY8=
|
||||
github.com/nxadm/tail v1.4.11 h1:8feyoE3OzPrcshW5/MJ4sGESc5cqmGkGCWlco4l0bqY=
|
||||
@@ -2170,10 +2174,6 @@ modernc.org/tcl v1.15.1/go.mod h1:aEjeGJX2gz1oWKOLDVZ2tnEWLUrIn8H+GFu+akoDhqs=
|
||||
modernc.org/token v1.0.1/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
|
||||
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
|
||||
modernc.org/z v1.7.0/go.mod h1:hVdgNMh8ggTuRG1rGU8x+xGRFfiQUIAw0ZqlPy8+HyQ=
|
||||
moq.dev/moq v0.7.27 h1:MICXMveLyoFBPj6Tu10oLH8NMhkYaDOa7oe6ay0RQ/c=
|
||||
moq.dev/moq v0.7.27/go.mod h1:zdCa+pXvmwXg+qTXgVJzO9jX7GbI7R/AFGwWOx1BKg0=
|
||||
moq.dev/moq-ffi v0.4.27 h1:PCXoyW1QTUN2XnM4pkAIBGpOAFjhm1OmCyPfPHszKPc=
|
||||
moq.dev/moq-ffi v0.4.27/go.mod h1:a+uEvcnNtY3e9T4/3UqjHBnyYHNLY8rZp+/m7tmL5n8=
|
||||
nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50=
|
||||
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
|
||||
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
"github.com/kerberos-io/agent/machinery/src/cloud/livemoq"
|
||||
"github.com/kerberos-io/agent/machinery/src/models"
|
||||
"github.com/kerberos-io/agent/machinery/src/packets"
|
||||
"moq.dev/moq"
|
||||
"github.com/moq-dev/moq-go/moq"
|
||||
log "github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
@@ -292,7 +292,7 @@ func publishLiveStreamMoQ(ctx context.Context, config liveMoQConfig) error {
|
||||
}
|
||||
defer finishBroadcast()
|
||||
|
||||
stream, err := broadcast.PublishVideo(moq.VideoFormatAvc3, nil)
|
||||
stream, err := broadcast.PublishMedia("avc3", nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create H.264 media stream: %w", err)
|
||||
}
|
||||
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
|
||||
"github.com/kerberos-io/agent/machinery/src/cloud/livemoq"
|
||||
"github.com/kerberos-io/agent/machinery/src/models"
|
||||
"moq.dev/moq"
|
||||
"github.com/moq-dev/moq-go/moq"
|
||||
)
|
||||
|
||||
func TestCloseLiveMoQPublisherFinishesBeforeClosingClient(t *testing.T) {
|
||||
@@ -155,7 +155,7 @@ func TestNativeMoQClientCloseStopsActiveWriter(t *testing.T) {
|
||||
server.Close()
|
||||
t.Fatal(err)
|
||||
}
|
||||
stream, err := broadcast.PublishVideo(moq.VideoFormatAvc3, nil)
|
||||
stream, err := broadcast.PublishMedia("avc3", nil)
|
||||
if err != nil {
|
||||
client.Close()
|
||||
server.Close()
|
||||
|
||||
Reference in New Issue
Block a user