Compare commits

..
Author SHA1 Message Date
Cédric Verstraeten 9d1d735928 Report sanitized tus upload creation errors
Extract tus and backend error codes from bounded response bodies to improve failure diagnostics without exposing sensitive messages. Add regression coverage for sanitization.
2026-10-10 19:32:08 +02:00
Cédric Verstraeten 7418aedd91 Fall back on persistent tus creation failures
Use legacy uploads after repeated server errors only if no tus upload was established. Preserve client-error handling and add coverage for fallback through Hub authorization.
2026-10-10 19:28:49 +02:00
33 changed files with 511 additions and 2831 deletions
-66
View File
@@ -76,60 +76,6 @@ There are a myriad of cameras out there (USB, IP and other cameras), and it migh
1. [What is new?](#what-is-new)
1. [Contributors](#contributors)
### Direct S3 recording layout
Direct S3 uploads store recordings as
`<s3.username>/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4`. The timestamp is the
recording's Unix timestamp in milliseconds, and the date is its UTC date, not the upload
date. Timestamp and device key come from the recording's `.metadata` upload
checkpoint, which stores the timestamp in milliseconds. The uploader also
preserves checkpoint duration, FPS (when available), and encryption status as S3
object metadata. The device key must be non-empty and contain no path separators.
Retried uploads use the same object key, even after midnight. Millisecond
precision keeps recordings starting within the same second distinct.
This replaces the previous `<s3.username>/<filename>.mp4` layout for new uploads.
Existing objects are not moved; consumers of direct S3 uploads must support both
layouts when reading historical recordings. Updated Kerberos Vault deployments
use the same filename and dated layout under the authenticated account directory
for both direct Vault uploads and Hub-proxied uploads.
Only historical recordings with missing, empty, or FPS-only queue markers fall
back to the legacy filename format and configured Agent key. A structured
checkpoint takes precedence over both the filename and current configured key;
an invalid or unreadable checkpoint is reported as an upload error, not silently
replaced with filename-derived values.
For older structured checkpoints, missing camera name, region, and change count
are recovered from a validated legacy filename. Populated checkpoint fields
take precedence, and the checkpoint timestamp and device key remain authoritative.
The shorter filename cannot be decoded by legacy filename-only consumers.
Integrations must preserve structured recording metadata, including the device
key (`productid`) and recording timestamp (`event-timestamp`), as well as the full
object key.
### Local recording layout
New local recordings are stored under
`data/recordings/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4`, using the recording's UTC
start date and Unix-millisecond timestamp. For example:
`data/recordings/2026/01/02/1767355200123-device-key.mp4`.
The filename no longer embeds the camera name, region, change count, or duration,
and it does not change when the recording is finalized.
Those attributes, together with FPS and encryption status, are captured in the
`.metadata` upload checkpoint and sent as structured HTTP/tus metadata.
Existing flat recordings and dated recordings with older long filenames remain
supported without a migration. Recording lists, day filters, playback, uploaders,
and automatic cleanup handle both filename formats and layouts. Upload queue
checkpoints remain flat in `data/cloud/`. Successful-upload removal also resolves
both local layouts. Deploy the updated Vault alongside Agent for canonical
`<account-directory>/yyyy/mm/dd/<timestamp>-<deviceKey>.mp4` storage keys and
metadata-driven Hub integration events.
Directory decryption also processes dated subdirectories and writes each output
to a sibling `decrypted/` directory; those outputs are excluded from recording
lists and automatic cleanup.
## Quickstart - Docker
The easiest way to get your Kerberos Agent up and running is to use our public image on [Docker hub](https://hub.docker.com/r/kerberos/agent). Once you have selected a specific tag, run `docker` command below, which will open the web interface of your Kerberos agent on port `80`, and off you go. For a more configurable and persistent deployment have a look at [Running and automating a Kerberos Agent](#running-and-automating-a-kerberos-agent).
@@ -384,7 +330,6 @@ See [RTSPS and TLS certificates](README-RTSPS-TLS.md) for the complete Bosch UI,
| `AGENT_CAPTURE_IPCAMERA_ONVIF_XADDR` | ONVIF endpoint/address running on the camera. | "" |
| `AGENT_CAPTURE_IPCAMERA_ONVIF_USERNAME` | ONVIF username to authenticate against. | "" |
| `AGENT_CAPTURE_IPCAMERA_ONVIF_PASSWORD` | ONVIF password to authenticate against. | "" |
| `AGENT_CAPTURE_IPCAMERA_ONVIF_CHANNEL` | 1-based channel on multi-channel ONVIF devices (encoders, NVRs) used for PTZ, presets and focus. | "0" (first profile) |
| `AGENT_CAPTURE_MOTION` | Toggle for enabling or disabling motion. | "true" |
| `AGENT_CAPTURE_LIVEVIEW` | Toggle for enabling or disabling liveview. | "true" |
| `AGENT_CAPTURE_SNAPSHOTS` | Toggle for enabling or disabling snapshot generation. | "true" |
@@ -465,17 +410,6 @@ Smaller chunks provide more frequent resumable checkpoints but create more HTTP
requests. Larger chunks reduce request overhead but require more data to be
retransmitted when a request fails.
An HTTP 500 during resumable creation is a server/provider failure, not proof
that the recording filename is invalid. Creation errors include up to 4 KiB of
the server response, and the final retry error retains the latest failure cause.
Created upload sessions remain usable if only the optional creation response
body is interrupted; the successful response's `Location` is retained.
Check the corresponding Hub/Vault logs and storage-provider configuration.
As a temporary diagnostic workaround, set `AGENT_DISABLE_RESUMABLE_UPLOAD=true`
and restart Agent to use ordinary uploads. This does not change the recording
filename or remove its structured metadata, but interrupted uploads restart
from the beginning. Remove the override after fixing the provider issue.
### Custom metadata headers for Kerberos Vault
Use `AGENT_KERBEROSVAULT_CUSTOM_HEADERS` to attach your own metadata (for
-47
View File
@@ -10,58 +10,11 @@ import (
"github.com/kerberos-io/agent/machinery/src/models"
)
func TestPickRecordingToCleanupAcrossDatedDirectories(t *testing.T) {
for _, legacyMarker := range []bool{false, true} {
t.Run(map[bool]string{false: "metadata", true: "legacy"}[legacyMarker], func(t *testing.T) {
recordingsDir, cloudDir := newCleanupDirs(t)
pendingName := "2026/01/01/oldest_pending.mp4"
safeName := "2026/01/02/uploaded.mp4"
writeRecording(t, recordingsDir, pendingName, 30)
if legacyMarker {
if err := os.WriteFile(filepath.Join(cloudDir, filepath.Base(pendingName)), nil, 0o644); err != nil {
t.Fatal(err)
}
} else {
markPending(t, cloudDir, pendingName)
}
writeRecording(t, recordingsDir, safeName, 20)
writeRecording(t, recordingsDir, "flat_uploaded.mp4", 10)
name, pending, err := pickRecordingToCleanup(recordingsDir, cloudDir)
if err != nil || pending || name != safeName {
t.Fatalf("cleanup = (%q, %v, %v), want (%q, false, nil)", name, pending, err, safeName)
}
})
}
}
func TestCleanupDatedPendingRecordingRemovesFlatMarker(t *testing.T) {
recordingsDir, cloudDir := newCleanupDirs(t)
name := "2026/01/01/pending.mp4"
if err := os.MkdirAll(filepath.Dir(filepath.Join(recordingsDir, name)), 0o755); err != nil {
t.Fatal(err)
}
writeSizedRecording(t, recordingsDir, name, 2*1000*1000)
markPending(t, cloudDir, name)
config := &models.Configuration{Config: models.Config{AutoClean: "true", MaxDirectorySize: 1}}
CleanupRecordingDirectory(filepath.Dir(filepath.Dir(recordingsDir)), config)
for _, filePath := range []string{
filepath.Join(recordingsDir, name),
filepath.Join(cloudDir, models.RecordingUploadMetadataFileName(name)),
} {
if _, err := os.Stat(filePath); !os.IsNotExist(err) {
t.Fatalf("expected %s to be removed, got %v", filePath, err)
}
}
}
// writeRecording creates a file under recordingsDir and sets its modtime so the
// tests can control the "oldest" ordering deterministically.
func writeRecording(t *testing.T, recordingsDir, name string, ageMinutes int) {
t.Helper()
full := filepath.Join(recordingsDir, name)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatalf("mkdir recording directory: %v", err)
}
if err := os.WriteFile(full, []byte("data"), 0o644); err != nil {
t.Fatalf("write recording %s: %v", name, err)
}
+161 -35
View File
@@ -53,16 +53,13 @@ func publishRecordingState(mqttClient mqtt.Client, hubKey string, configuration
}
}
func recordingUploadMetadata(name string, config models.Config, timestamp int64, mp4Video *video.MP4, encrypted bool, coordinates string, changes int) models.RecordingUploadMetadata {
func recordingUploadMetadata(name, deviceKey string, timestamp int64, mp4Video *video.MP4, encrypted bool) models.RecordingUploadMetadata {
metadata := models.RecordingUploadMetadata{
FileName: filepath.Base(name),
DeviceKey: config.Key,
Timestamp: timestamp,
Duration: mp4Video.VideoTotalDuration,
Encrypted: encrypted,
DeviceName: config.Name,
RegionCoordinates: coordinates,
NumberOfChanges: strconv.Itoa(changes),
FileName: filepath.Base(name),
DeviceKey: deviceKey,
Timestamp: timestamp,
Duration: mp4Video.VideoTotalDuration,
Encrypted: encrypted,
}
value := mp4Video.AverageFPS()
if value > 0 && value <= 240 && !math.IsInf(value, 0) && !math.IsNaN(value) {
@@ -193,7 +190,7 @@ func CleanupRecordingDirectory(configDirectory string, configuration *models.Con
return
}
if err := utils.RemoveRecording(recordingsDirectory, name); err != nil {
if err := os.Remove(recordingsDirectory + "/" + name); err != nil {
log.Info("HandleRecordStream: something went wrong, " + err.Error())
return
}
@@ -278,7 +275,7 @@ func defaultReserveMB(totalMB int64) int64 {
// overall with pending=true, signalling the caller that it is about to drop an
// un-uploaded recording as a last resort.
func pickRecordingToCleanup(recordingsDirectory, cloudDirectory string) (string, bool, error) {
entries, err := utils.ReadRecordingDirectory(recordingsDirectory)
entries, err := os.ReadDir(recordingsDirectory)
if err != nil {
return "", false, err
}
@@ -287,7 +284,14 @@ func pickRecordingToCleanup(recordingsDirectory, cloudDirectory string) (string,
var oldestSafeTime, oldestAnyTime time.Time
for _, entry := range entries {
modTime := entry.ModTime()
if entry.IsDir() {
continue
}
info, err := entry.Info()
if err != nil || !info.Mode().IsRegular() {
continue
}
modTime := info.ModTime()
if oldestAnyName == "" || modTime.Before(oldestAnyTime) {
oldestAnyName = entry.Name()
@@ -445,6 +449,39 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
// Cleanup muxer
start = false
// Update the name of the recording with the duration.
// We will update the name of the recording with the duration in milliseconds.
if mp4Video.VideoTotalDuration > 0 {
duration := mp4Video.VideoTotalDuration
// Update the name with the duration in milliseconds.
startRecordingSeconds := startRecording / 1000 // convert to seconds
startRecordingMilliseconds := startRecording % 1000 // convert to milliseconds
s := strconv.FormatInt(startRecordingSeconds, 10) + "_" +
strconv.Itoa(len(strconv.FormatInt(startRecordingMilliseconds, 10))) + "-" +
strconv.FormatInt(startRecordingMilliseconds, 10) + "_" +
config.Name + "_" +
"0-0-0-0" + "_" + // region coordinates, we
"-1" + "_" + // token
strconv.FormatInt(int64(duration), 10) // + "_" + // duration of recording
//utils.VERSION // version of the agent
oldName := name
name = s + ".mp4"
fullName = configDirectory + "/data/recordings/" + name
log.Info("capture.main.HandleRecordStream(motiondetection): renamed file from: " + oldName + " to: " + name)
// Rename the file to the new name.
err := os.Rename(
configDirectory+"/data/recordings/"+oldName,
configDirectory+"/data/recordings/"+s+".mp4")
if err != nil {
log.Error("capture.main.HandleRecordStream(motiondetection): error renaming file: " + err.Error())
}
} else {
log.Info("capture.main.HandleRecordStream(continuous): no video data recorded, not renaming file.")
}
encrypted := false
// Check if we need to encrypt the recording.
if config.Encryption != nil && config.Encryption.Enabled == "true" && config.Encryption.Recordings == "true" && config.Encryption.SymmetricKey != "" {
@@ -469,7 +506,7 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
}
}
queueRecordingForUpload(configDirectory, recordingUploadMetadata(name, config, startRecording, mp4Video, encrypted, "0-0-0-0", -1))
queueRecordingForUpload(configDirectory, recordingUploadMetadata(name, config.Key, startRecording, mp4Video, encrypted))
recordingStatus = "idle"
@@ -494,17 +531,29 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
start = true
// timestamp_microseconds_instanceName_regionCoordinates_numberOfChanges_token
// 1564859471_6-474162_oprit_577-283-727-375_1153_27.mp4
// - Timestamp
// - Size + - + microseconds
// - device
// - Region
// - Number of changes
// - Token
startRecording = pkt.CurrentTime
name, err = utils.RecordingFileName(startRecording, config.Key)
if err == nil {
fullName, err = utils.CreateRecordingPath(configDirectory+"/data/recordings", name, time.UnixMilli(startRecording))
}
if err != nil {
log.WithError(err).Error("capture: could not create continuous recording path")
start = false
pkt = nextPkt
continue
}
startRecordingSeconds := startRecording / 1000 // convert to seconds
startRecordingMilliseconds := startRecording % 1000 // convert to milliseconds
s := strconv.FormatInt(startRecordingSeconds, 10) + "_" + // start timestamp in seconds
strconv.Itoa(len(strconv.FormatInt(startRecordingMilliseconds, 10))) + "-" + // length of milliseconds
strconv.FormatInt(startRecordingMilliseconds, 10) + "_" + // milliseconds
config.Name + "_" + // device name
"0-0-0-0" + "_" + // region coordinates, we will not use this for continuous recording
"0" + "_" + // token
"0" + "_" //+ // duration of recording in milliseconds
//utils.VERSION // version of the agent
name = s + ".mp4"
fullName = configDirectory + "/data/recordings/" + name
// Running...
log.Info("capture.main.HandleRecordStream(continuous): recording started")
@@ -560,6 +609,39 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
// Cleanup muxer
start = false
// Update the name of the recording with the duration.
// We will update the name of the recording with the duration in milliseconds.
if mp4Video.VideoTotalDuration > 0 {
duration := mp4Video.VideoTotalDuration
// Update the name with the duration in milliseconds.
startRecordingSeconds := startRecording / 1000 // convert to seconds
startRecordingMilliseconds := startRecording % 1000 // convert to milliseconds
s := strconv.FormatInt(startRecordingSeconds, 10) + "_" +
strconv.Itoa(len(strconv.FormatInt(startRecordingMilliseconds, 10))) + "-" +
strconv.FormatInt(startRecordingMilliseconds, 10) + "_" +
config.Name + "_" +
"0-0-0-0" + "_" + // region coordinates, we
"-1" + "_" + // token
strconv.FormatInt(int64(duration), 10) // + "_" + // duration of recording
//utils.VERSION // version of the agent
oldName := name
name = s + ".mp4"
fullName = configDirectory + "/data/recordings/" + name
log.Info("capture.main.HandleRecordStream(motiondetection): renamed file from: " + oldName + " to: " + name)
// Rename the file to the new name.
err := os.Rename(
configDirectory+"/data/recordings/"+oldName,
configDirectory+"/data/recordings/"+s+".mp4")
if err != nil {
log.Error("capture.main.HandleRecordStream(motiondetection): error renaming file: " + err.Error())
}
} else {
log.Info("capture.main.HandleRecordStream(continuous): no video data recorded, not renaming file.")
}
encrypted := false
// Check if we need to encrypt the recording.
if config.Encryption != nil && config.Encryption.Enabled == "true" && config.Encryption.Recordings == "true" && config.Encryption.SymmetricKey != "" {
@@ -584,7 +666,7 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
}
}
queueRecordingForUpload(configDirectory, recordingUploadMetadata(name, config, startRecording, mp4Video, encrypted, "0-0-0-0", -1))
queueRecordingForUpload(configDirectory, recordingUploadMetadata(name, config.Key, startRecording, mp4Video, encrypted))
recordingStatus = "idle"
@@ -639,7 +721,17 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
continue
}
displayTimeSeconds := displayTime / 1000
// timestamp_microseconds_instanceName_regionCoordinates_numberOfChanges_token
// 1564859471_6-474162_oprit_577-283-727-375_1153_27.mp4
// - Timestamp
// - Size + - + microseconds
// - device
// - Region
// - Number of changes
// - Token
displayTimeSeconds := displayTime / 1000 // convert to seconds
displayTimeMilliseconds := displayTime % 1000 // convert to milliseconds
motionRectangleString := "0-0-0-0"
if motion.Rectangle.X != 0 || motion.Rectangle.Y != 0 ||
motion.Rectangle.Width != 0 || motion.Rectangle.Height != 0 {
@@ -650,15 +742,17 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
// Get the number of changes from the motion detection.
numberOfChanges := motion.NumberOfChanges
name, err := utils.RecordingFileName(displayTime, config.Key)
var fullName string
if err == nil {
fullName, err = utils.CreateRecordingPath(configDirectory+"/data/recordings", name, time.UnixMilli(displayTime))
}
if err != nil {
log.WithError(err).Error("capture: could not create motion recording path")
continue
}
s := strconv.FormatInt(displayTimeSeconds, 10) + "_" + // start timestamp in seconds
strconv.Itoa(len(strconv.FormatInt(displayTimeMilliseconds, 10))) + "-" + // length of milliseconds
strconv.FormatInt(displayTimeMilliseconds, 10) + "_" + // milliseconds
config.Name + "_" + // device name
motionRectangleString + "_" + // region coordinates, we will not use this for continuous recording
strconv.Itoa(numberOfChanges) + "_" + // number of changes
"0" // + "_" + // duration of recording in milliseconds
//utils.VERSION // version of the agent
name := s + ".mp4"
fullName := configDirectory + "/data/recordings/" + name
// Running...
log.Info("capture.main.HandleRecordStream(motiondetection): recording started (" + name + ")" + " at " + strconv.FormatInt(displayTimeSeconds, 10) + " unix")
@@ -786,6 +880,38 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
communication.TrySendMotion(models.MotionDataPartial{Timestamp: time.Now().Unix(), NumberOfChanges: 100000000})
}
// Update the name of the recording with the duration.
// We will update the name of the recording with the duration in milliseconds.
if mp4Video.VideoTotalDuration > 0 {
duration := mp4Video.VideoTotalDuration
// Update the name with the duration in milliseconds.
s := strconv.FormatInt(displayTimeSeconds, 10) + "_" +
strconv.Itoa(len(strconv.FormatInt(displayTimeMilliseconds, 10))) + "-" +
strconv.FormatInt(displayTimeMilliseconds, 10) + "_" +
config.Name + "_" +
motionRectangleString + "_" +
strconv.Itoa(numberOfChanges) + "_" + // number of changes
strconv.FormatInt(int64(duration), 10) // + "_" + // duration of recording in milliseconds
//utils.VERSION // version of the agent
oldName := name
name = s + ".mp4"
fullName = configDirectory + "/data/recordings/" + name
log.Info("capture.main.HandleRecordStream(motiondetection): renamed file from: " + oldName + " to: " + name)
// Rename the file to the new name.
err := os.Rename(
configDirectory+"/data/recordings/"+oldName,
configDirectory+"/data/recordings/"+s+".mp4")
if err != nil {
log.Error("capture.main.HandleRecordStream(motiondetection): error renaming file: " + err.Error())
}
} else {
log.Info("capture.main.HandleRecordStream(motiondetection): no video data recorded, not renaming file.")
}
encrypted := false
// Check if we need to encrypt the recording.
if config.Encryption != nil && config.Encryption.Enabled == "true" && config.Encryption.Recordings == "true" && config.Encryption.SymmetricKey != "" {
@@ -810,7 +936,7 @@ func HandleRecordStream(queue *packets.Queue, configDirectory string, configurat
}
}
queueRecordingForUpload(configDirectory, recordingUploadMetadata(name, config, displayTime, mp4Video, encrypted, motionRectangleString, numberOfChanges))
queueRecordingForUpload(configDirectory, recordingUploadMetadata(name, config.Key, displayTime, mp4Video, encrypted))
// Clean up the recording directory if necessary.
CleanupRecordingDirectory(configDirectory, configuration)
+3 -5
View File
@@ -42,11 +42,10 @@ func TestQueueRecordingForUploadStoresFinalizedMetadata(t *testing.T) {
}
mp4Video := &video.MP4{VideoTotalDuration: 20452, SampleCount: 613}
config := models.Config{Key: "device-key", Name: "camera-name"}
metadata := recordingUploadMetadata("1785934709414-device-key.mp4", config, 1785934709414, mp4Video, true, "1-2-3-4", 57)
metadata := recordingUploadMetadata("recording.mp4", "device-key", 1785934709414, mp4Video, true)
queueRecordingForUpload(configDirectory, metadata)
got, err := os.ReadFile(filepath.Join(configDirectory, "data", "cloud", "1785934709414-device-key.metadata"))
got, err := os.ReadFile(filepath.Join(configDirectory, "data", "cloud", "recording.metadata"))
if err != nil {
t.Fatalf("read upload marker: %v", err)
}
@@ -55,8 +54,7 @@ func TestQueueRecordingForUploadStoresFinalizedMetadata(t *testing.T) {
t.Fatalf("decode upload marker: %v", err)
}
expectedFPS := mp4Video.AverageFPS()
if stored.FileName != "1785934709414-device-key.mp4" || stored.DeviceKey != "device-key" || stored.Timestamp != 1785934709414 || stored.Duration != 20452 || math.Abs(stored.FPS-expectedFPS) > 1e-9 || !stored.Encrypted ||
stored.DeviceName != "camera-name" || stored.RegionCoordinates != "1-2-3-4" || stored.NumberOfChanges != "57" {
if stored.FileName != "recording.mp4" || stored.DeviceKey != "device-key" || stored.Timestamp != 1785934709414 || stored.Duration != 20452 || math.Abs(stored.FPS-expectedFPS) > 1e-9 || !stored.Encrypted {
t.Fatalf("upload marker = %+v", stored)
}
if stored.FPS == math.Floor(stored.FPS) {
+2 -7
View File
@@ -125,7 +125,7 @@ func HandleUpload(configDirectory string, configuration *models.Configuration, c
// Check if we need to remove the original recording
// removeAfterUpload is set to false by default
if config.RemoveAfterUpload != "false" {
err := utils.RemoveRecording(configDirectory+"/data/recordings", fileName)
err := os.Remove(configDirectory + "/data/recordings/" + fileName)
if err != nil {
log.Error("HandleUpload: " + err.Error())
}
@@ -319,7 +319,6 @@ func heartbeatONVIFCameraKey(camera models.IPCamera) string {
camera.ONVIFXAddr,
camera.ONVIFUsername,
camera.ONVIFPassword,
strconv.Itoa(camera.ONVIFChannel),
}, "\x00")
}
@@ -414,7 +413,7 @@ func (state *heartbeatONVIFState) refreshStaticPayload(device *goonvif.Device) h
payload.panTilt = "true"
}
presets, err := heartbeatGetPresetsFromDevice(device, state.cameraConfiguration.ONVIFChannel)
presets, err := heartbeatGetPresetsFromDevice(device)
if err == nil && len(presets) > 0 {
payload.presets = "true"
presetsList, marshalErr := json.Marshal(presets)
@@ -1380,10 +1379,6 @@ func HandleRealtimeProcessing(processingCursor *packets.QueueCursor, configurati
// Check if we need to enable the realtime processing
if config.RealtimeProcessing == "true" {
if mqttClient == nil {
log.Warn("cloud.RealtimeProcessing(): stopping as MQTT client is not configured.")
return
}
hubKey := ""
if config.Cloud == "s3" && config.S3 != nil && config.S3.Publickey != "" {
+4 -4
View File
@@ -296,7 +296,7 @@ func TestHeartbeatONVIFPayloadCachesStaticAndReusesLoopSubscription(t *testing.T
ptzFunctionCalls++
return nil, true, true
}
heartbeatGetPresetsFromDevice = func(*goonvif.Device, int) ([]models.OnvifActionPreset, error) {
heartbeatGetPresetsFromDevice = func(*goonvif.Device) ([]models.OnvifActionPreset, error) {
presetCalls++
return []models.OnvifActionPreset{{Name: "Lobby", Token: "1"}}, nil
}
@@ -397,7 +397,7 @@ func TestHeartbeatONVIFPayloadRefreshesAfterCameraConfigChange(t *testing.T) {
heartbeatGetPTZFunctionsFromDevice = func(goonvifptz.GetConfigurationsResponse) ([]string, bool, bool) {
return nil, false, true
}
heartbeatGetPresetsFromDevice = func(*goonvif.Device, int) ([]models.OnvifActionPreset, error) {
heartbeatGetPresetsFromDevice = func(*goonvif.Device) ([]models.OnvifActionPreset, error) {
presetCalls++
return nil, nil
}
@@ -477,7 +477,7 @@ func TestHeartbeatONVIFPayloadRetriesStaticFetchFailuresWithoutReconnect(t *test
heartbeatGetPTZFunctionsFromDevice = func(goonvifptz.GetConfigurationsResponse) ([]string, bool, bool) {
return nil, true, true
}
heartbeatGetPresetsFromDevice = func(*goonvif.Device, int) ([]models.OnvifActionPreset, error) {
heartbeatGetPresetsFromDevice = func(*goonvif.Device) ([]models.OnvifActionPreset, error) {
presetCalls++
if presetCalls == 1 {
return nil, errors.New("temporary preset failure")
@@ -618,7 +618,7 @@ func TestHeartbeatONVIFPayloadReconnectFailureInvalidatesCache(t *testing.T) {
heartbeatGetPTZFunctionsFromDevice = func(goonvifptz.GetConfigurationsResponse) ([]string, bool, bool) {
return nil, true, false
}
heartbeatGetPresetsFromDevice = func(*goonvif.Device, int) ([]models.OnvifActionPreset, error) {
heartbeatGetPresetsFromDevice = func(*goonvif.Device) ([]models.OnvifActionPreset, error) {
presetCalls++
return []models.OnvifActionPreset{{Name: "Preset", Token: "1"}}, nil
}
+4 -2
View File
@@ -5,13 +5,13 @@ package cloud
import (
"bytes"
"errors"
"os"
"github.com/dropbox/dropbox-sdk-go-unofficial/v6/dropbox"
"github.com/dropbox/dropbox-sdk-go-unofficial/v6/dropbox/files"
"github.com/dropbox/dropbox-sdk-go-unofficial/v6/dropbox/users"
"github.com/gin-gonic/gin"
"github.com/kerberos-io/agent/machinery/src/models"
"github.com/kerberos-io/agent/machinery/src/utils"
log "github.com/sirupsen/logrus"
)
@@ -37,12 +37,14 @@ func UploadDropbox(configuration *models.Configuration, fileName string) (bool,
// Upload to Dropbox
log.Info("UploadDropbox: Uploading to Dropbox")
log.Info("UploadDropbox: Upload started for " + fileName)
fullname := "data/recordings/" + fileName
dConfig := dropbox.Config{
Token: token,
LogLevel: dropbox.LogInfo, // if needed, set the desired logging level. Default is off
}
file, err := utils.OpenRecording("data/recordings", fileName)
file, err := os.OpenFile(fullname, os.O_RDWR, 0755)
if file != nil {
defer func() {
if cerr := file.Close(); cerr != nil {
+3 -2
View File
@@ -9,7 +9,6 @@ import (
"os"
"github.com/kerberos-io/agent/machinery/src/models"
"github.com/kerberos-io/agent/machinery/src/utils"
log "github.com/sirupsen/logrus"
)
@@ -79,8 +78,10 @@ func UploadKerberosHub(configuration *models.Configuration, fileName string) (bo
}).Info("Resumable Hub upload unavailable; using legacy upload")
}
fullname := "data/recordings/" + fileName
// Check if we still have the file otherwise we abort the request.
file, err := utils.OpenRecording("data/recordings", fileName)
file, err := os.OpenFile(fullname, os.O_RDWR, 0755)
if file != nil {
defer file.Close()
}
+6 -18
View File
@@ -10,7 +10,6 @@ import (
"time"
"github.com/kerberos-io/agent/machinery/src/models"
"github.com/kerberos-io/agent/machinery/src/utils"
log "github.com/sirupsen/logrus"
)
@@ -36,23 +35,10 @@ func UploadKerberosVault(configuration *models.Configuration, fileName string) (
// This can happen when the file was already removed (e.g. cleanup, or an
// earlier successful upload). Skip it so the watcher drops the marker
// instead of retrying indefinitely.
filePath, err := utils.ResolveRecordingPath("data/recordings", fileName)
info, err := os.Stat("data/recordings/" + fileName)
if err != nil {
if os.IsNotExist(err) {
log.Info("UploadKerberosVault: skipping " + fileName + ", file doesn't exist anymore")
return false, false, nil
}
log.WithError(err).Error("UploadKerberosVault: could not resolve recording path")
return false, true, err
}
info, err := os.Stat(filePath)
if err != nil {
if os.IsNotExist(err) {
log.Info("UploadKerberosVault: skipping " + fileName + ", file doesn't exist anymore")
return false, false, nil
}
log.WithError(err).Error("UploadKerberosVault: could not stat recording")
return false, true, err
log.Info("UploadKerberosVault: skipping " + fileName + ", file doesn't exist anymore")
return false, false, nil
}
if info.Size() == 0 {
log.Warn("UploadKerberosVault: skipping " + fileName + ", recording is empty")
@@ -196,7 +182,9 @@ func uploadVaultLegacy(vault models.KStorage, publicKey, deviceKey, fileName, la
if err != nil {
return false, false, "", err
}
file, err := utils.OpenRecording("data/recordings", fileName)
fullname := "data/recordings/" + fileName
file, err := os.Open(fullname)
if file != nil {
defer file.Close()
}
@@ -1,58 +0,0 @@
package cloud
import (
"testing"
"github.com/kerberos-io/agent/machinery/src/models"
log "github.com/sirupsen/logrus"
"github.com/sirupsen/logrus/hooks/test"
)
func TestHandleRealtimeProcessingWithoutMQTTClient(t *testing.T) {
tests := []struct {
name string
config models.Config
wantWarning bool
}{
{
name: "enabled without Hub credentials",
config: models.Config{RealtimeProcessing: "true"},
wantWarning: true,
},
{
name: "disabled",
config: models.Config{RealtimeProcessing: "false"},
},
{
name: "offline",
config: models.Config{RealtimeProcessing: "true", Offline: "true"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
hook := test.NewGlobal()
defer hook.Reset()
configuration := &models.Configuration{Config: tt.config}
// No cursor or decoder should be accessed without an MQTT client.
HandleRealtimeProcessing(nil, configuration, nil, nil, nil)
warnings := 0
for _, entry := range hook.AllEntries() {
if entry.Level == log.WarnLevel {
warnings++
if entry.Message != "cloud.RealtimeProcessing(): stopping as MQTT client is not configured." {
t.Errorf("unexpected warning: %s", entry.Message)
}
}
}
wantWarnings := 0
if tt.wantWarning {
wantWarnings = 1
}
if warnings != wantWarnings {
t.Fatalf("warnings = %d, want %d", warnings, wantWarnings)
}
})
}
}
+2 -30
View File
@@ -2,7 +2,6 @@ package cloud
import (
"encoding/json"
"fmt"
"math"
"net/http"
"os"
@@ -11,16 +10,12 @@ import (
"strings"
"github.com/kerberos-io/agent/machinery/src/models"
log "github.com/sirupsen/logrus"
)
const recordingFPSHeader = "X-Kerberos-Storage-Fps"
const recordingDurationHeader = "X-Kerberos-Storage-Duration"
const recordingTimestampHeader = "X-Kerberos-Storage-Timestamp"
const recordingEncryptedHeader = "X-Kerberos-Storage-Encrypted"
const recordingDeviceNameHeader = "X-Kerberos-Storage-DeviceName"
const recordingRegionHeader = "X-Kerberos-Storage-RegionCoordinates"
const recordingChangesHeader = "X-Kerberos-Storage-NumberOfChanges"
// queuedRecordingFPS reads the FPS snapshot written into the upload marker
// when the recording was finalized. Historical empty markers intentionally
@@ -66,14 +61,6 @@ func decodeRecordingUploadMetadata(value []byte) (models.RecordingUploadMetadata
}
func readRecordingUploadMetadata(fileName string) ([]byte, bool) {
value, err := readRecordingUploadMetadataFile(fileName)
if err != nil && !os.IsNotExist(err) {
log.WithError(err).Error("Failed to read recording upload metadata")
}
return value, err == nil
}
func readRecordingUploadMetadataFile(fileName string) ([]byte, error) {
markerNames := []string{
models.RecordingUploadMetadataFileName(fileName),
filepath.Base(fileName),
@@ -81,13 +68,10 @@ func readRecordingUploadMetadataFile(fileName string) ([]byte, error) {
for _, markerName := range markerNames {
value, err := os.ReadFile(filepath.Join("data", "cloud", markerName))
if err == nil {
return value, nil
}
if !os.IsNotExist(err) {
return nil, fmt.Errorf("read recording upload metadata %s: %w", markerName, err)
return value, true
}
}
return nil, os.ErrNotExist
return nil, false
}
func setQueuedRecordingMetadataHeaders(header http.Header, fileName string) {
@@ -95,18 +79,6 @@ func setQueuedRecordingMetadataHeaders(header http.Header, fileName string) {
header.Set(recordingFPSHeader, fps)
}
if metadata, ok := queuedRecordingMetadata(fileName); ok {
if metadata.DeviceKey != "" {
header.Set("X-Kerberos-Storage-Device", metadata.DeviceKey)
}
if metadata.DeviceName != "" {
header.Set(recordingDeviceNameHeader, metadata.DeviceName)
}
if metadata.RegionCoordinates != "" {
header.Set(recordingRegionHeader, metadata.RegionCoordinates)
}
if metadata.NumberOfChanges != "" {
header.Set(recordingChangesHeader, metadata.NumberOfChanges)
}
if metadata.Duration > 0 {
header.Set(recordingDurationHeader, strconv.FormatUint(metadata.Duration, 10))
}
+36 -155
View File
@@ -2,20 +2,14 @@ package cloud
import (
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"math"
"net/http"
"net/url"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/kerberos-io/agent/machinery/src/models"
"github.com/kerberos-io/agent/machinery/src/utils"
"github.com/minio/minio-go/v6"
log "github.com/sirupsen/logrus"
)
@@ -24,6 +18,15 @@ func UploadS3(configuration *models.Configuration, fileName string) (bool, bool,
config := configuration.Config
// timestamp_microseconds_instanceName_regionCoordinates_numberOfChanges_token
// 1564859471_6-474162_oprit_577-283-727-375_1153_27.mp4
// - Timestamp
// - Size + - + microseconds
// - device
// - Region
// - Number of changes
// - Token
if config.S3 == nil {
errorMessage := "UploadS3: Uploading Failed, as no settings found"
log.Error(errorMessage)
@@ -68,14 +71,24 @@ func UploadS3(configuration *models.Configuration, fileName string) (bool, bool,
s3Client.SetCustomTransport(transport)
}
objectName, userMetadata, err := s3RecordingUploadDetails(config, fileName, aws_access_key_id)
if err != nil {
log.WithError(err).Error("UploadS3: invalid recording upload metadata")
return false, true, err
fileParts := strings.Split(fileName, "_")
if len(fileParts) == 1 {
errorMessage := "UploadS3: " + fileName + " is not a valid name."
log.Error(errorMessage)
return false, true, errors.New(errorMessage)
}
deviceKey := config.Key
startRecording, _ := strconv.ParseInt(fileParts[0], 10, 64)
devicename := fileParts[2]
coordinates := fileParts[3]
//numberOfChanges := fileParts[4]
token, _ := strconv.Atoi(fileParts[5])
log.Info("UploadS3: Upload started for " + fileName)
file, err := utils.OpenRecording("data/recordings", fileName)
fullname := "data/recordings/" + fileName
file, err := os.OpenFile(fullname, os.O_RDWR, 0755)
if file != nil {
defer file.Close()
}
@@ -94,13 +107,23 @@ func UploadS3(configuration *models.Configuration, fileName string) (bool, bool,
}
n, err := s3Client.PutObject(config.S3.Bucket,
objectName,
config.S3.Username+"/"+fileName,
file,
fileInfo.Size(),
minio.PutObjectOptions{
ContentType: "video/mp4",
StorageClass: "ONEZONE_IA",
UserMetadata: userMetadata,
UserMetadata: map[string]string{
"event-timestamp": strconv.FormatInt(startRecording, 10),
"event-microseconds": deviceKey,
"event-instancename": devicename,
"event-regioncoordinates": coordinates,
"event-numberofchanges": deviceKey,
"event-token": strconv.Itoa(token),
"productid": deviceKey,
"publickey": aws_access_key_id,
"uploadtime": "now",
},
})
if err != nil {
@@ -112,145 +135,3 @@ func UploadS3(configuration *models.Configuration, fileName string) (bool, bool,
return true, true, nil
}
}
func s3RecordingObjectName(username, deviceKey string, startRecording int64) string {
return username + "/" + time.UnixMilli(startRecording).UTC().Format("2006/01/02") +
"/" + strconv.FormatInt(startRecording, 10) + "-" + deviceKey + ".mp4"
}
func s3RecordingUploadDetails(config models.Config, fileName, publicKey string) (string, map[string]string, error) {
value, err := readRecordingUploadMetadataFile(fileName)
if err != nil && !os.IsNotExist(err) {
return "", nil, err
}
deviceKey := config.Key
var startRecording int64
userMetadata := map[string]string{
"publickey": publicKey,
"uploadtime": "now",
}
marker := strings.TrimSpace(string(value))
_, fpsErr := strconv.ParseFloat(marker, 64)
if err == nil && marker != "" && fpsErr != nil {
var metadata models.RecordingUploadMetadata
if err := json.Unmarshal(value, &metadata); err != nil {
return "", nil, fmt.Errorf("decode recording upload metadata for %s: %w", fileName, err)
}
if metadata.FileName != "" && metadata.FileName != filepath.Base(fileName) {
return "", nil, fmt.Errorf("recording upload metadata filename does not match %s", fileName)
}
if metadata.Timestamp <= 0 {
return "", nil, fmt.Errorf("invalid recording timestamp in upload metadata for %s", fileName)
}
deviceKey = metadata.DeviceKey
startRecording = metadata.Timestamp
userMetadata["event-instancename"] = config.Name
if metadata.DeviceName == "" || metadata.RegionCoordinates == "" || metadata.NumberOfChanges == "" {
if description, ok := s3LegacyRecordingDescription(fileName); ok {
for key, value := range description {
userMetadata[key] = value
}
}
}
if metadata.DeviceName != "" {
userMetadata["event-instancename"] = metadata.DeviceName
}
if metadata.RegionCoordinates != "" {
userMetadata["event-regioncoordinates"] = metadata.RegionCoordinates
}
if metadata.NumberOfChanges != "" {
userMetadata["event-numberofchanges"] = metadata.NumberOfChanges
}
userMetadata["event-microseconds"] = strconv.FormatInt(metadata.Timestamp%1000*1000, 10)
userMetadata["duration"] = strconv.FormatUint(metadata.Duration, 10)
userMetadata["encrypted"] = strconv.FormatBool(metadata.Encrypted)
if metadata.FPS > 0 && metadata.FPS <= 240 {
userMetadata["fps"] = strconv.FormatFloat(metadata.FPS, 'f', -1, 64)
}
} else {
// Empty and FPS-only markers predate structured recording metadata.
fileParts := strings.Split(fileName, "_")
if len(fileParts) < 6 {
return "", nil, fmt.Errorf("%s is not a valid legacy recording name", fileName)
}
startRecording, err = strconv.ParseInt(fileParts[0], 10, 64)
if err != nil {
return "", nil, fmt.Errorf("invalid recording timestamp in %s: %w", fileName, err)
}
if startRecording <= 0 || startRecording > math.MaxInt64/1000 {
return "", nil, fmt.Errorf("invalid recording timestamp in %s", fileName)
}
startRecording *= 1000
_, precision, found := strings.Cut(fileParts[1], "-")
if found {
fraction, err := strconv.ParseInt(precision, 10, 64)
if err != nil || fraction < 0 || fraction >= 1000000 {
return "", nil, fmt.Errorf("invalid recording precision in %s", fileName)
}
if len(precision) > 3 {
fraction /= 1000
}
startRecording += fraction
}
token, _ := strconv.Atoi(fileParts[5])
userMetadata["event-microseconds"] = deviceKey
userMetadata["event-instancename"] = fileParts[2]
userMetadata["event-regioncoordinates"] = fileParts[3]
userMetadata["event-numberofchanges"] = deviceKey
userMetadata["event-token"] = strconv.Itoa(token)
}
if deviceKey == "" || strings.ContainsAny(deviceKey, "/\\") {
return "", nil, errors.New("missing or invalid recording device key")
}
if startRecording <= 0 {
return "", nil, fmt.Errorf("invalid recording timestamp for %s", fileName)
}
userMetadata["event-timestamp"] = strconv.FormatInt(startRecording/1000, 10)
userMetadata["productid"] = deviceKey
return s3RecordingObjectName(config.S3.Username, deviceKey, startRecording), userMetadata, nil
}
func s3LegacyRecordingDescription(fileName string) (map[string]string, bool) {
base := filepath.Base(fileName)
if filepath.Ext(base) != ".mp4" {
return nil, false
}
parts := strings.Split(strings.TrimSuffix(base, ".mp4"), "_")
if len(parts) != 6 || parts[2] == "" {
return nil, false
}
seconds, err := strconv.ParseInt(parts[0], 10, 64)
if err != nil || seconds <= 0 || seconds > math.MaxInt64/1000 {
return nil, false
}
digits, fraction, found := strings.Cut(parts[1], "-")
precision, err := strconv.Atoi(digits)
if !found || err != nil || precision < 1 || precision > 6 || len(fraction) != precision {
return nil, false
}
if _, err := strconv.ParseUint(fraction, 10, 64); err != nil {
return nil, false
}
coordinates := strings.Split(parts[3], "-")
if len(coordinates) != 4 {
return nil, false
}
for _, coordinate := range coordinates {
if _, err := strconv.ParseUint(coordinate, 10, 64); err != nil {
return nil, false
}
}
changes, err := strconv.ParseInt(parts[4], 10, 64)
if err != nil || changes < -1 {
return nil, false
}
if _, err := strconv.ParseUint(parts[5], 10, 64); err != nil {
return nil, false
}
return map[string]string{
"event-instancename": parts[2],
"event-regioncoordinates": parts[3],
"event-numberofchanges": parts[4],
}, true
}
-328
View File
@@ -1,328 +0,0 @@
package cloud
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"strconv"
"strings"
"testing"
"time"
"github.com/kerberos-io/agent/machinery/src/models"
)
func TestS3RecordingUploadDetailsUsesCheckpoint(t *testing.T) {
for _, fileName := range []string{
"1767355200123-checkpoint-key.mp4",
"1564859471_6-474162_old-camera_0-0-0-0_1153_27.mp4",
} {
t.Run(fileName, func(t *testing.T) {
withRecording(t, fileName, []byte("recording"))
recorded := time.Date(2026, time.January, 2, 12, 0, 0, 123000000, time.UTC)
metadata := models.RecordingUploadMetadata{
FileName: fileName,
DeviceKey: "checkpoint-key",
Timestamp: recorded.UnixMilli(),
Duration: 20452,
FPS: 29.97,
Encrypted: true,
DeviceName: "checkpoint-camera",
RegionCoordinates: "1-2-3-4",
NumberOfChanges: "57",
}
payload, err := json.Marshal(metadata)
if err != nil {
t.Fatal(err)
}
withQueuedRecordingFPS(t, fileName, string(payload))
config := models.Config{
Key: "changed-config-key",
Name: "camera-name",
S3: &models.S3{Username: "user"},
}
objectName, userMetadata, err := s3RecordingUploadDetails(config, fileName, "public-key")
if err != nil {
t.Fatal(err)
}
wantObjectName := "user/2026/01/02/" + strconv.FormatInt(recorded.UnixMilli(), 10) + "-checkpoint-key.mp4"
if objectName != wantObjectName {
t.Fatalf("object name = %q, want %q", objectName, wantObjectName)
}
wantMetadata := map[string]string{
"publickey": "public-key",
"uploadtime": "now",
"event-instancename": "checkpoint-camera",
"event-regioncoordinates": "1-2-3-4",
"event-numberofchanges": "57",
"event-microseconds": "123000",
"event-timestamp": strconv.FormatInt(recorded.Unix(), 10),
"productid": "checkpoint-key",
"duration": "20452",
"fps": "29.97",
"encrypted": "true",
}
if !reflect.DeepEqual(userMetadata, wantMetadata) {
t.Fatalf("metadata = %#v, want %#v", userMetadata, wantMetadata)
}
})
}
}
func TestS3RecordingUploadDetailsBackfillsLegacyCheckpointDescriptions(t *testing.T) {
const legacyName = "1564859471_6-474162_recorded-camera_1-2-3-4_1153_27.mp4"
tests := []struct {
name string
fileName string
camera string
region string
changes string
wantCamera string
wantRegion string
wantChanges string
}{
{"pre-upgrade checkpoint", legacyName, "", "", "", "recorded-camera", "1-2-3-4", "1153"},
{"checkpoint camera", legacyName, "snapshot-camera", "", "", "snapshot-camera", "1-2-3-4", "1153"},
{"checkpoint region", legacyName, "", "5-6-7-8", "", "recorded-camera", "5-6-7-8", "1153"},
{"checkpoint zero changes", legacyName, "", "", "0", "recorded-camera", "1-2-3-4", "0"},
{"dated reference", "2019/08/03/" + legacyName, "", "", "", "recorded-camera", "1-2-3-4", "1153"},
{"continuous recording", "1564859471_3-474_recorded-camera_0-0-0-0_-1_27.mp4", "", "", "", "recorded-camera", "0-0-0-0", "-1"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
withRecording(t, filepath.Base(tt.fileName), []byte("recording"))
metadata := models.RecordingUploadMetadata{
FileName: filepath.Base(tt.fileName),
DeviceKey: "checkpoint-key",
Timestamp: 1767355200123,
Duration: 20452,
FPS: 29.97,
Encrypted: true,
DeviceName: tt.camera,
RegionCoordinates: tt.region,
NumberOfChanges: tt.changes,
}
payload, err := json.Marshal(metadata)
if err != nil {
t.Fatal(err)
}
withQueuedRecordingFPS(t, tt.fileName, string(payload))
config := models.Config{Key: "current-key", Name: "current-camera", S3: &models.S3{Username: "user"}}
objectName, got, err := s3RecordingUploadDetails(config, tt.fileName, "public-key")
if err != nil {
t.Fatal(err)
}
if want := "user/2026/01/02/1767355200123-checkpoint-key.mp4"; objectName != want {
t.Fatalf("object name = %q, want %q", objectName, want)
}
want := map[string]string{
"publickey": "public-key",
"uploadtime": "now",
"event-instancename": tt.wantCamera,
"event-regioncoordinates": tt.wantRegion,
"event-numberofchanges": tt.wantChanges,
"event-microseconds": "123000",
"event-timestamp": "1767355200",
"productid": "checkpoint-key",
"duration": "20452",
"fps": "29.97",
"encrypted": "true",
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("metadata = %#v, want %#v", got, want)
}
})
}
}
func TestS3RecordingUploadDetailsDoesNotBackfillInvalidLegacyNames(t *testing.T) {
for _, fileName := range []string{
"1767355200123-checkpoint-key.mp4",
"invalid_6-474162_camera_0-0-0-0_1153_27.mp4",
"1564859471_6-invalid_camera_0-0-0-0_1153_27.mp4",
"1564859471_3-474162_camera_0-0-0-0_1153_27.mp4",
"1564859471_6-474162__0-0-0-0_1153_27.mp4",
"1564859471_6-474162_camera_invalid_1153_27.mp4",
"1564859471_6-474162_camera_0-0-0-0_invalid_27.mp4",
"1564859471_6-474162_camera_0-0-0-0_1153_invalid.mp4",
"1564859471_6-474162_camera_0-0-0-0_1153_27_extra.mp4",
} {
t.Run(fileName, func(t *testing.T) {
withRecording(t, fileName, []byte("recording"))
metadata := models.RecordingUploadMetadata{
FileName: fileName, DeviceKey: "checkpoint-key", Timestamp: 1767355200123, Duration: 20452,
}
payload, err := json.Marshal(metadata)
if err != nil {
t.Fatal(err)
}
withQueuedRecordingFPS(t, fileName, string(payload))
config := models.Config{Key: "current-key", Name: "current-camera", S3: &models.S3{Username: "user"}}
objectName, got, err := s3RecordingUploadDetails(config, fileName, "public-key")
if err != nil {
t.Fatal(err)
}
if objectName != "user/2026/01/02/1767355200123-checkpoint-key.mp4" ||
got["event-instancename"] != "current-camera" || got["event-regioncoordinates"] != "" ||
got["event-numberofchanges"] != "" || got["event-timestamp"] != "1767355200" ||
got["productid"] != "checkpoint-key" {
t.Fatalf("unexpected fallback for invalid legacy name: %q, %#v", objectName, got)
}
})
}
}
func TestS3RecordingUploadDetailsLegacyMarkers(t *testing.T) {
for _, marker := range []string{"missing", "empty", "fps-only", "old-marker-name"} {
t.Run(marker, func(t *testing.T) {
const fileName = "1564859471_6-474162_camera_0-0-0-0_1153_27.mp4"
withRecording(t, fileName, []byte("recording"))
switch marker {
case "empty":
withQueuedRecordingFPS(t, fileName, "")
case "fps-only", "old-marker-name":
withQueuedRecordingFPS(t, fileName, "29.97")
if marker == "old-marker-name" {
err := os.Rename(filepath.Join("data/cloud", models.RecordingUploadMetadataFileName(fileName)),
filepath.Join("data/cloud", fileName))
if err != nil {
t.Fatal(err)
}
}
}
config := models.Config{Key: "legacy-key", S3: &models.S3{Username: "user"}}
objectName, metadata, err := s3RecordingUploadDetails(config, fileName, "public-key")
if err != nil {
t.Fatal(err)
}
if want := "user/2019/08/03/1564859471474-legacy-key.mp4"; objectName != want {
t.Fatalf("object name = %q, want %q", objectName, want)
}
if metadata["productid"] != "legacy-key" || metadata["event-instancename"] != "camera" ||
metadata["event-regioncoordinates"] != "0-0-0-0" || metadata["event-timestamp"] != "1564859471" {
t.Fatalf("unexpected legacy metadata: %#v", metadata)
}
})
}
}
func TestS3RecordingUploadDetailsRejectsInvalidCheckpoint(t *testing.T) {
tests := []struct {
name string
marker string
want string
}{
{"malformed JSON", `{"timestamp":`, "decode"},
{"missing timestamp", `{"device_key":"camera"}`, "timestamp"},
{"missing device key", `{"timestamp":1787015373000}`, "device key"},
{"invalid device key", `{"timestamp":1787015373000,"device_key":"camera/key"}`, "device key"},
{"mismatched filename", `{"timestamp":1787015373000,"device_key":"camera","filename":"other.mp4"}`, "filename"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
const fileName = "1564859471_6-474162_camera_0-0-0-0_1153_27.mp4"
withRecording(t, fileName, []byte("recording"))
withQueuedRecordingFPS(t, fileName, tt.marker)
config := models.Config{Key: "legacy-key", S3: &models.S3{Username: "user"}}
_, _, err := s3RecordingUploadDetails(config, fileName, "public-key")
if err == nil || !strings.Contains(err.Error(), tt.want) {
t.Fatalf("error = %v, want error containing %q instead of legacy fallback", err, tt.want)
}
})
}
}
func TestS3RecordingUploadDetailsReportsCheckpointReadError(t *testing.T) {
const fileName = "1564859471_6-474162_camera_0-0-0-0_1153_27.mp4"
withRecording(t, fileName, []byte("recording"))
if err := os.MkdirAll(filepath.Join("data/cloud", models.RecordingUploadMetadataFileName(fileName)), 0o755); err != nil {
t.Fatal(err)
}
config := models.Config{Key: "legacy-key", S3: &models.S3{Username: "user"}}
_, _, err := s3RecordingUploadDetails(config, fileName, "public-key")
if err == nil || !strings.Contains(err.Error(), "read recording upload metadata") {
t.Fatalf("error = %v, want checkpoint read error instead of legacy fallback", err)
}
}
func TestS3RecordingObjectName(t *testing.T) {
tests := []struct {
name string
recorded time.Time
directory string
}{
{
name: "zero padded month and day",
recorded: time.Date(2026, time.January, 2, 12, 0, 0, 0, time.UTC),
directory: "2026/01/02",
},
{
name: "UTC date across year boundary",
recorded: time.Date(2026, time.January, 1, 1, 0, 0, 0, time.FixedZone("UTC+2", 2*60*60)),
directory: "2025/12/31",
},
{
name: "leap day",
recorded: time.Date(2024, time.February, 29, 23, 59, 59, 0, time.UTC),
directory: "2024/02/29",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
const deviceKey = "camera-key-123"
want := "user/" + tt.directory + "/" + strconv.FormatInt(tt.recorded.UnixMilli(), 10) + "-" + deviceKey + ".mp4"
for attempt := 0; attempt < 2; attempt++ {
if got := s3RecordingObjectName("user", deviceKey, tt.recorded.UnixMilli()); got != want {
t.Fatalf("s3RecordingObjectName() = %q, want %q", got, want)
}
}
})
}
}
func TestUploadS3RejectsInvalidRecordingNames(t *testing.T) {
configuration := &models.Configuration{
Config: models.Config{
Key: "camera-key-123",
S3: &models.S3{
Publickey: "test-public-key",
Secretkey: "test-secret-key",
Region: "eu-west-1",
},
},
}
for _, fileName := range []string{
"recording.mp4",
"1787015373_camera.mp4",
"invalid_6-474162_camera_0-0-0-0_1153_27.mp4",
} {
t.Run(fileName, func(t *testing.T) {
uploaded, configured, err := UploadS3(configuration, fileName)
if uploaded || !configured || err == nil {
t.Fatalf("UploadS3() = (%v, %v, %v), want (false, true, error)", uploaded, configured, err)
}
})
}
}
func TestUploadS3RejectsInvalidDeviceKeys(t *testing.T) {
for _, deviceKey := range []string{"", "camera/key", `camera\key`} {
t.Run(deviceKey, func(t *testing.T) {
configuration := &models.Configuration{
Config: models.Config{
Key: deviceKey,
S3: &models.S3{
Publickey: "test-public-key",
Secretkey: "test-secret-key",
Region: "eu-west-1",
},
},
}
uploaded, configured, err := UploadS3(configuration, "1787015373_6-474162_camera_0-0-0-0_1153_27.mp4")
if uploaded || !configured || err == nil || !strings.Contains(err.Error(), "device key") {
t.Fatalf("UploadS3() = (%v, %v, %v), want (false, true, device key error)", uploaded, configured, err)
}
})
}
}
+39 -39
View File
@@ -10,13 +10,13 @@ import (
"net/url"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
"github.com/kerberos-io/agent/machinery/src/models"
"github.com/kerberos-io/agent/machinery/src/utils"
log "github.com/sirupsen/logrus"
)
@@ -50,6 +50,12 @@ func resumableUploadsEnabled() bool {
const tusDefaultChunkSize int64 = 8 << 20 // 8 MiB (>= S3 multipart minimum part size)
const tusProgressBucketPercent int64 = 10
const tusErrorBodyLimit int64 = 16 << 10
var (
tusErrorCodePattern = regexp.MustCompile(`\bERR_[A-Z0-9_]+\b`)
tusBackendErrorPattern = regexp.MustCompile(`\bapi error ([A-Za-z][A-Za-z0-9._-]{0,63}):`)
)
// tusChunkSize returns the number of bytes to send per PATCH request. It
// defaults to tusDefaultChunkSize (8 MiB) and can be overridden with the
@@ -107,7 +113,7 @@ func logTusUploadProgress(label string, offset, size int64, loggedBucket *int64)
// creation request; it is empty on HEAD/PATCH/DELETE.
type tusHeaderFunc func(h http.Header, fileName string)
// runTusUpload performs a resumable (tus) upload of a flat or dated recording.
// runTusUpload performs a resumable (tus) upload of data/recordings/<fileName>
// to baseURL, sending target-specific authentication/routing headers via
// setHeaders on every request. It encapsulates the create/resume/chunk/finalize
// state machine shared by the Kerberos Vault (direct) and Kerberos Hub (proxied)
@@ -121,7 +127,9 @@ type tusHeaderFunc func(h http.Header, fileName string)
// should fall back to the legacy single-POST upload (older deployments).
// - body: a short message for logging.
func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tusHeaderFunc) (uploaded bool, responded bool, supported bool, body string, err error) {
file, ferr := utils.OpenRecording("data/recordings", fileName)
fullname := "data/recordings/" + fileName
file, ferr := os.Open(fullname)
if file != nil {
defer file.Close()
}
@@ -143,6 +151,7 @@ func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tu
sidecar := tusSidecarPath(fileName, slot)
uploadURL := loadTusResumeState(sidecar, baseURL)
uploadEstablished := uploadURL != ""
const maxAttempts = 4
restartedAfterComplete := false
@@ -164,7 +173,6 @@ func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tu
// and loop forever, wedging the upload worker on one recording and saturating the
// uplink.
highWaterOffset := int64(0)
var lastErr error
for attempt := 0; attempt < maxAttempts; attempt++ {
// (1) Ensure we have an active upload URL, creating one if needed.
@@ -172,7 +180,6 @@ func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tu
created, status, cerr := tusCreate(client, baseURL, size, metadata, setHeaders, fileName)
lastStatus = status
if cerr != nil {
lastErr = cerr
if status == http.StatusNotFound || status == http.StatusMethodNotAllowed || status == http.StatusNotImplemented {
// The vault does not implement tus; let the caller fall back.
return false, false, false, "", cerr
@@ -182,6 +189,7 @@ func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tu
continue
}
uploadURL = created
uploadEstablished = true
saveTusResumeState(sidecar, tusResumeState{UploadURL: uploadURL, VaultURI: baseURL, Size: size})
}
@@ -189,7 +197,6 @@ func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tu
offset, status, herr := tusHead(client, uploadURL, setHeaders)
lastStatus = status
if herr != nil {
lastErr = herr
if status == http.StatusNotFound || status == http.StatusGone {
// The upload expired/was removed server-side; start over.
removeTusResumeState(sidecar)
@@ -246,7 +253,6 @@ func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tu
newOffset, status, respBody, perr := tusPatch(client, uploadURL, offset, patchLen, file, setHeaders)
lastStatus = status
if perr != nil {
lastErr = perr
if status >= 400 {
// Definitive rejection (e.g. provider push failed during finalize).
// Re-evaluate via HEAD on the next iteration to decide retry/restart.
@@ -293,12 +299,15 @@ func runTusUpload(baseURL, metadata, fileName, label, slot string, setHeaders tu
// report responded=false so the caller keeps the recording queued and retries
// later instead of consuming its retry budget and entering the long back-off
// timeout.
message := label + ": resumable upload did not complete after retries"
err = errors.New(message)
if lastErr != nil {
err = fmt.Errorf("%s: %w", message, lastErr)
supported = true
if !uploadEstablished && lastStatus >= http.StatusInternalServerError {
// A persistent server failure before creation means no upload URL was
// issued and no recording bytes were sent. Falling back to the legacy
// endpoint is therefore safe and preserves uploads while the server-side
// tus/provider path is unavailable.
supported = false
}
return false, lastStatus > 0, true, "resumable upload did not complete after retries", err
return false, lastStatus > 0, supported, "resumable upload did not complete after retries", errors.New(label + ": resumable upload did not complete after retries")
}
// uploadVaultResumable uploads a recording directly to a Kerberos Vault using
@@ -366,12 +375,6 @@ func addRecordingTusMetadata(values map[string]string, fileName string) {
if !ok {
return
}
if metadata.DeviceKey != "" {
values["device"] = metadata.DeviceKey
}
values["device_name"] = metadata.DeviceName
values["region_coordinates"] = metadata.RegionCoordinates
values["number_of_changes"] = metadata.NumberOfChanges
if metadata.Duration > 0 {
values["duration"] = strconv.FormatUint(metadata.Duration, 10)
}
@@ -404,29 +407,17 @@ func tusCreate(client *http.Client, baseURL string, size int64, metadata string,
if err != nil {
return "", 0, err
}
if resp.StatusCode != http.StatusCreated {
const maxErrorBodyBytes = 4096
body, readErr := io.ReadAll(io.LimitReader(resp.Body, maxErrorBodyBytes+1))
if readErr != nil {
return "", resp.StatusCode, fmt.Errorf("unexpected status creating upload: %s; read error response: %w", resp.Status, readErr)
}
truncated := len(body) > maxErrorBodyBytes
if truncated {
body = body[:maxErrorBodyBytes]
}
message := strings.TrimSpace(string(body))
if truncated {
message += " [truncated]"
}
if message != "" {
return "", resp.StatusCode, fmt.Errorf("unexpected status creating upload: %s: %q", resp.Status, message)
body, _ := io.ReadAll(io.LimitReader(resp.Body, tusErrorBodyLimit))
_, _ = io.Copy(io.Discard, resp.Body)
details := tusErrorDetails(body)
if details != "" {
return "", resp.StatusCode, fmt.Errorf("unexpected status creating upload: %s (%s)", resp.Status, details)
}
return "", resp.StatusCode, fmt.Errorf("unexpected status creating upload: %s", resp.Status)
}
// Location identifies the created upload; its response body is optional.
if _, err := io.Copy(io.Discard, resp.Body); err != nil {
log.WithError(err).WithField("component", "tus").Debug("Could not drain optional upload creation response body")
}
_, _ = io.Copy(io.Discard, resp.Body)
location := resp.Header.Get("Location")
if location == "" {
return "", resp.StatusCode, errors.New("missing Location header in create response")
@@ -434,6 +425,17 @@ func tusCreate(client *http.Client, baseURL string, size int64, metadata string,
return resolveTusLocation(baseURL, location), resp.StatusCode, nil
}
func tusErrorDetails(body []byte) string {
var details []string
if code := tusErrorCodePattern.Find(body); len(code) != 0 {
details = append(details, string(code))
}
if match := tusBackendErrorPattern.FindSubmatch(body); len(match) == 2 {
details = append(details, "backend="+string(match[1]))
}
return strings.Join(details, ", ")
}
// tusHead performs the tus "offset" request (HEAD) and returns the current
// server-side upload offset.
func tusHead(client *http.Client, uploadURL string, setHeaders tusHeaderFunc) (int64, int, error) {
@@ -532,7 +534,6 @@ func setVaultTusHeaders(h http.Header, vault models.KStorage, publicKey, deviceK
h.Set("X-Kerberos-Storage-Capture", "IPCamera")
if fileName != "" {
h.Set("X-Kerberos-Storage-FileName", fileName)
setQueuedRecordingMetadataHeaders(h, fileName)
}
}
@@ -549,7 +550,6 @@ func setHubTusHeaders(h http.Header, config *models.Config, fileName string) {
h.Set("X-Kerberos-Storage-Capture", "IPCamera")
if fileName != "" {
h.Set("X-Kerberos-Storage-FileName", fileName)
setQueuedRecordingMetadataHeaders(h, fileName)
}
}
+138 -219
View File
@@ -14,231 +14,12 @@ import (
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/kerberos-io/agent/machinery/src/models"
"github.com/kerberos-io/agent/machinery/src/utils"
)
func TestResumableEOFRestartPreservesLatestFailure(t *testing.T) {
previousDelay := tusBackoffBaseDelay
tusBackoffBaseDelay = 0
t.Cleanup(func() { tusBackoffBaseDelay = previousDelay })
const fileName = "1767355200123-device-key.mp4"
withRecording(t, fileName, []byte("recording"))
fake := newFakeTus()
fake.failFinalize = 1
var heads atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead && heads.Add(1) <= 2 {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
fake.ServeHTTP(w, r)
}))
defer server.Close()
uploaded, responded, supported, _, err := uploadVaultResumable(testVault(server.URL), "public", "device-key", fileName, "test", "primary")
if uploaded || !responded || !supported || err == nil ||
!strings.Contains(err.Error(), "unexpected status on PATCH: 502 Bad Gateway") ||
strings.Contains(err.Error(), "503 Service Unavailable") {
t.Fatalf("upload = %t/%t/%t, %v; want the latest finalization failure after EOF restart", uploaded, responded, supported, err)
}
patches, _ := fake.patchCounts()
if heads.Load() != 4 || fake.createCount() != 1 || patches != 1 ||
len(fake.requestsForMethod(http.MethodDelete)) != 1 {
t.Fatalf("unexpected retry sequence: heads=%d, creates=%d, patches=%d", heads.Load(), fake.createCount(), patches)
}
}
func TestResumableCreationKeepsLocationAfterBodyReadError(t *testing.T) {
previousDelay := tusBackoffBaseDelay
tusBackoffBaseDelay = 0
t.Cleanup(func() { tusBackoffBaseDelay = previousDelay })
const fileName = "1767355200123-device-key.mp4"
payload := []byte("recording")
withRecording(t, fileName, payload)
fake := newFakeTus()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPost {
w.Header().Set("Content-Length", "100")
fake.ServeHTTP(w, r)
_, _ = io.WriteString(w, "short")
return
}
fake.ServeHTTP(w, r)
}))
defer server.Close()
uploaded, responded, supported, _, err := uploadVaultResumable(testVault(server.URL), "public", "device-key", fileName, "test", "primary")
if !uploaded || !responded || !supported || err != nil {
t.Fatalf("upload = %t/%t/%t, %v; want success despite the interrupted optional creation body", uploaded, responded, supported, err)
}
if fake.createCount() != 1 || fake.totalBytes() != int64(len(payload)) {
t.Fatalf("created %d slots and uploaded %d bytes; want one reused slot and %d bytes", fake.createCount(), fake.totalBytes(), len(payload))
}
}
func TestTusCreateStillRequiresLocationAfterBodyReadError(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Length", "100")
w.WriteHeader(http.StatusCreated)
_, _ = io.WriteString(w, "short")
}))
defer server.Close()
location, status, err := tusCreate(server.Client(), server.URL, 8, "",
func(http.Header, string) {}, "1767355200123-device-key.mp4")
if location != "" || status != http.StatusCreated || err == nil ||
!strings.Contains(err.Error(), "missing Location") {
t.Fatalf("creation = %q, %d, %v; want missing Location error", location, status, err)
}
}
func TestTusCreateReportsBoundedServerErrors(t *testing.T) {
for _, tc := range []struct {
name string
body string
want string
}{
{"server error", "ERR_INTERNAL_SERVER_ERROR: A region must be set when sending requests to S3.\n", "A region must be set"},
{"empty error", "", "500 Internal Server Error"},
{"large error", strings.Repeat("x", 10000), "[truncated]"},
} {
t.Run(tc.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(w, tc.body)
}))
defer server.Close()
location, status, err := tusCreate(server.Client(), server.URL, 8, "",
func(http.Header, string) {}, "1767355200123-device-key.mp4")
if location != "" || status != http.StatusInternalServerError || err == nil ||
!strings.Contains(err.Error(), tc.want) || len(err.Error()) > 4300 {
t.Fatalf("creation = %q, %d, %v; want bounded server error", location, status, err)
}
})
}
}
func TestResumableCreateFailureRetainsServerCause(t *testing.T) {
previousDelay := tusBackoffBaseDelay
tusBackoffBaseDelay = 0
t.Cleanup(func() { tusBackoffBaseDelay = previousDelay })
for _, integration := range []string{"vault", "hub"} {
t.Run(integration, func(t *testing.T) {
const fileName = "1767355200123-device-key.mp4"
withRecording(t, fileName, []byte("recording"))
var requests atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
if r.Method != http.MethodPost || r.URL.Path != tusUploadPath {
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
w.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(w, "ERR_INTERNAL_SERVER_ERROR: A region must be set when sending requests to S3.")
}))
defer server.Close()
var uploaded, responded, supported bool
var err error
if integration == "vault" {
uploaded, responded, supported, _, err = uploadVaultResumable(testVault(server.URL), "public", "device-key", fileName, "test", "primary")
} else {
uploaded, responded, supported, _, err = uploadHubResumable(testHubConfig(server.URL), fileName, "test", "hub")
}
if uploaded || !responded || !supported || err == nil ||
!strings.Contains(err.Error(), "resumable upload did not complete after retries") ||
!strings.Contains(err.Error(), "A region must be set") {
t.Fatalf("upload = %t/%t/%t, %v; want original server cause after retries", uploaded, responded, supported, err)
}
if requests.Load() != 4 {
t.Fatalf("creation attempts = %d, want 4 without legacy fallback", requests.Load())
}
if _, err := os.Stat(filepath.Join("data/recordings", fileName)); err != nil {
t.Fatalf("failed upload must retain the local recording: %v", err)
}
})
}
}
func TestUploadKerberosVaultReadsDatedLocalRecordings(t *testing.T) {
previousCount, previousTimeout := kstorageRetryCount, kstorageRetryTimeout
t.Cleanup(func() { kstorageRetryCount, kstorageRetryTimeout = previousCount, previousTimeout })
for _, transport := range []string{"resumable", "legacy"} {
t.Run(transport, func(t *testing.T) {
const fileName = "1564859471123-device-key.mp4"
payload := bytes.Repeat([]byte("video"), 1024)
withRecording(t, fileName, payload)
datedPath, err := utils.CreateRecordingPath("data/recordings", fileName, time.Unix(1564859471, 0))
if err != nil {
t.Fatal(err)
}
if err := os.Rename(filepath.Join("data/recordings", fileName), datedPath); err != nil {
t.Fatal(err)
}
withQueuedRecordingFPS(t, fileName, `{"filename":"1564859471123-device-key.mp4","device_key":"device-key","timestamp":1564859471123,"duration":27,"fps":25,"device_name":"camera-name","region_coordinates":"1-2-3-4","number_of_changes":"57"}`)
fake := newFakeTus()
var legacyBody []byte
var legacyHeaders http.Header
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if transport == "resumable" {
fake.ServeHTTP(w, r)
return
}
body, readErr := io.ReadAll(r.Body)
if readErr != nil {
t.Error(readErr)
}
legacyBody = body
legacyHeaders = r.Header.Clone()
w.WriteHeader(http.StatusOK)
}))
defer server.Close()
if transport == "legacy" {
t.Setenv("AGENT_DISABLE_RESUMABLE_UPLOAD", "true")
} else {
t.Setenv("AGENT_DISABLE_RESUMABLE_UPLOAD", "false")
}
kstorageRetryCount, kstorageRetryTimeout = 0, 0
vault := testVault(server.URL)
config := &models.Configuration{Config: models.Config{
Key: "changed-config-key",
KStorage: &vault,
KStorageSecondary: &models.KStorage{},
}}
uploaded, configured, err := UploadKerberosVault(config, fileName)
if err != nil || !uploaded || !configured {
t.Fatalf("UploadKerberosVault() = (%v, %v, %v), want success", uploaded, configured, err)
}
if transport == "resumable" {
if got := fake.totalBytes(); got != int64(len(payload)) {
t.Fatalf("uploaded %d bytes, want %d", got, len(payload))
}
posts := fake.requestsForMethod(http.MethodPost)
if len(posts) != 1 {
t.Fatalf("got %d creation requests, want 1", len(posts))
}
metadata := decodeTusMetadata(posts[0].header.Get("Upload-Metadata"))
if metadata["filename"] != fileName || metadata["directory"] != "dir" || metadata["timestamp"] != "1564859471123" ||
metadata["device"] != "device-key" || metadata["device_name"] != "camera-name" ||
metadata["region_coordinates"] != "1-2-3-4" || metadata["number_of_changes"] != "57" {
t.Fatalf("unexpected upload metadata: %#v", metadata)
}
} else if !bytes.Equal(legacyBody, payload) ||
legacyHeaders.Get("X-Kerberos-Storage-FileName") != fileName ||
legacyHeaders.Get("X-Kerberos-Storage-Directory") != "dir" ||
legacyHeaders.Get("X-Kerberos-Storage-Device") != "device-key" ||
legacyHeaders.Get(recordingDeviceNameHeader) != "camera-name" ||
legacyHeaders.Get(recordingRegionHeader) != "1-2-3-4" ||
legacyHeaders.Get(recordingChangesHeader) != "57" ||
legacyHeaders.Get(recordingTimestampHeader) != "1564859471123" {
t.Fatalf("unexpected legacy upload: %d bytes, headers %#v", len(legacyBody), legacyHeaders)
}
})
}
}
// fakeUpload tracks the state of a single resumable upload on the fake server.
type fakeUpload struct {
size int64
@@ -265,6 +46,8 @@ type fakeTus struct {
// unsupported makes the creation endpoint return 404, simulating an older
// vault without a tus endpoint.
unsupported bool
// createStatus makes every creation request fail with the configured status.
createStatus int
// failFinalize causes the next N completing PATCH requests to return 502
// after storing the bytes, simulating a failed completion hook.
failFinalize int
@@ -353,6 +136,13 @@ func (s *fakeTus) ServeHTTP(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
return
}
if s.createStatus != 0 {
s.mu.Lock()
s.creates++
s.mu.Unlock()
w.WriteHeader(s.createStatus)
return
}
length, _ := strconv.ParseInt(r.Header.Get("Upload-Length"), 10, 64)
s.mu.Lock()
s.counter++
@@ -1059,6 +849,135 @@ func TestUploadHubResumable_Unsupported(t *testing.T) {
}
}
func TestUploadHubResumableFallsBackAfterCreateServerErrors(t *testing.T) {
srv := newFakeTus()
srv.createStatus = http.StatusInternalServerError
ts := httptest.NewServer(srv)
defer ts.Close()
fileName := "f.mp4"
withRecording(t, fileName, []byte("hello"))
oldDelay := tusBackoffBaseDelay
tusBackoffBaseDelay = time.Millisecond
t.Cleanup(func() { tusBackoffBaseDelay = oldDelay })
uploaded, responded, supported, _, err := uploadHubResumable(testHubConfig(ts.URL), fileName, "test", "hub")
if uploaded {
t.Fatal("uploaded = true, want false")
}
if !responded {
t.Fatal("responded = false, want true")
}
if supported {
t.Fatal("supported = true, want false so the caller uses the legacy upload")
}
if err == nil {
t.Fatal("error = nil, want resumable create failure")
}
if got := srv.createCount(); got != 4 {
t.Fatalf("create attempts = %d, want 4", got)
}
}
func TestUploadKerberosHubFallsBackThroughHubAfterTusCreateServerErrors(t *testing.T) {
fileName := "f.mp4"
payload := []byte("recording")
withRecording(t, fileName, payload)
t.Setenv("AGENT_DISABLE_RESUMABLE_UPLOAD", "")
oldDelay := tusBackoffBaseDelay
tusBackoffBaseDelay = time.Millisecond
t.Cleanup(func() { tusBackoffBaseDelay = oldDelay })
var tusCreates, legacyHeads, legacyPosts int
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodPost && r.URL.Path == tusUploadPath:
tusCreates++
w.WriteHeader(http.StatusInternalServerError)
case r.Method == http.MethodHead && r.URL.Path == "/storage/upload":
legacyHeads++
w.WriteHeader(http.StatusOK)
case r.Method == http.MethodPost && r.URL.Path == "/storage/upload":
legacyPosts++
got, err := io.ReadAll(r.Body)
if err != nil {
t.Errorf("read legacy upload: %v", err)
}
if !bytes.Equal(got, payload) {
t.Errorf("legacy upload body = %q, want %q", got, payload)
}
w.WriteHeader(http.StatusOK)
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
w.WriteHeader(http.StatusNotFound)
}
}))
defer server.Close()
config := testHubConfig(server.URL)
uploaded, responded, err := UploadKerberosHub(&models.Configuration{Config: *config}, fileName)
if err != nil {
t.Fatalf("UploadKerberosHub error: %v", err)
}
if !uploaded || !responded {
t.Fatalf("UploadKerberosHub = uploaded %v, responded %v; want true, true", uploaded, responded)
}
if tusCreates != 4 {
t.Errorf("tus create attempts = %d, want 4", tusCreates)
}
if legacyHeads != 1 {
t.Errorf("legacy Hub authorization requests = %d, want 1", legacyHeads)
}
if legacyPosts != 1 {
t.Errorf("legacy Hub upload requests = %d, want 1", legacyPosts)
}
}
func TestTusCreateReportsSanitizedServerErrorCodes(t *testing.T) {
const sensitiveMessage = "credential=do-not-log-this"
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(w, "ERR_INTERNAL_SERVER_ERROR: operation error S3: CreateMultipartUpload, api error AccessDenied: "+sensitiveMessage)
}))
defer server.Close()
_, status, err := tusCreate(server.Client(), server.URL, 8, "", func(http.Header, string) {}, "f.mp4")
if status != http.StatusInternalServerError {
t.Fatalf("status = %d, want %d", status, http.StatusInternalServerError)
}
if err == nil {
t.Fatal("error = nil, want create failure")
}
message := err.Error()
for _, want := range []string{"ERR_INTERNAL_SERVER_ERROR", "backend=AccessDenied"} {
if !strings.Contains(message, want) {
t.Errorf("error %q does not contain %q", message, want)
}
}
if strings.Contains(message, sensitiveMessage) {
t.Fatalf("error exposed backend message: %q", message)
}
}
func TestUploadHubResumableDoesNotFallbackAfterCreateClientError(t *testing.T) {
srv := newFakeTus()
srv.createStatus = http.StatusBadRequest
ts := httptest.NewServer(srv)
defer ts.Close()
fileName := "f.mp4"
withRecording(t, fileName, []byte("hello"))
oldDelay := tusBackoffBaseDelay
tusBackoffBaseDelay = time.Millisecond
t.Cleanup(func() { tusBackoffBaseDelay = oldDelay })
_, _, supported, _, _ := uploadHubResumable(testHubConfig(ts.URL), fileName, "test", "hub")
if !supported {
t.Fatal("supported = false for a client rejection, which must not bypass authorization via legacy fallback")
}
}
func TestEncodeTusMetadata(t *testing.T) {
got := encodeTusMetadata(map[string]string{
"b": "2",
+3 -5
View File
@@ -964,7 +964,7 @@ func GetDashboard(c *gin.Context, configDirectory string, configuration *models.
// All days stored in this agent.
days := []string{}
latestEvents := []models.Media{}
files, err := utils.ReadRecordingDirectory(recordingDirectory)
files, err := utils.ReadDirectory(recordingDirectory)
if err == nil {
events := utils.GetSortedDirectory(files)
@@ -975,8 +975,6 @@ func GetDashboard(c *gin.Context, configDirectory string, configuration *models.
var eventFilter models.EventFilter
eventFilter.NumberOfElements = 5
latestEvents = utils.GetMediaFormatted(events, recordingDirectory, configuration, eventFilter) // will get 5 latest recordings.
} else {
log.WithError(err).Error("Could not list recordings for Agent status")
}
c.JSON(200, gin.H{
@@ -1009,7 +1007,7 @@ func GetLatestEvents(c *gin.Context, configDirectory string, configuration *mode
eventFilter.NumberOfElements = 10
}
recordingDirectory := configDirectory + "/data/recordings"
files, err := utils.ReadRecordingDirectory(recordingDirectory)
files, err := utils.ReadDirectory(recordingDirectory)
if err == nil {
events := utils.GetSortedDirectory(files)
// We will get all recordings from the directory (as defined by the filter).
@@ -1038,7 +1036,7 @@ func GetLatestEvents(c *gin.Context, configDirectory string, configuration *mode
// @Success 200
func GetDays(c *gin.Context, configDirectory string, configuration *models.Configuration, communication *models.Communication) {
recordingDirectory := configDirectory + "/data/recordings"
files, err := utils.ReadRecordingDirectory(recordingDirectory)
files, err := utils.ReadDirectory(recordingDirectory)
if err == nil {
events := utils.GetSortedDirectory(files)
days := utils.GetDays(events, recordingDirectory, configuration)
-3
View File
@@ -394,9 +394,6 @@ func applyAgentEnvVars(configuration *models.Configuration, prefix string, apply
case "AGENT_CAPTURE_IPCAMERA_ONVIF_PASSWORD":
configuration.Config.Capture.IPCamera.ONVIFPassword = value
break
case "AGENT_CAPTURE_IPCAMERA_ONVIF_CHANNEL":
configuration.Config.Capture.IPCamera.ONVIFChannel, _ = strconv.Atoi(value)
break
/* Recording mode */
case "AGENT_CAPTURE_RECORDING":
-6
View File
@@ -12,7 +12,6 @@ type OnvifCredentials struct {
ONVIFXAddr string `json:"onvif_xaddr,omitempty" bson:"onvif_xaddr"`
ONVIFUsername string `json:"onvif_username,omitempty" bson:"onvif_username"`
ONVIFPassword string `json:"onvif_password,omitempty" bson:"onvif_password"`
ONVIFChannel int `json:"onvif_channel,omitempty" bson:"onvif_channel,omitempty"` // 1-based; 0 = first profile
}
type CameraStreams struct {
@@ -69,11 +68,6 @@ type OnvifZoom struct {
Zoom float64 `json:"zoom,omitempty" bson:"zoom"`
}
type OnvifFocus struct {
OnvifCredentials OnvifCredentials `json:"onvif_credentials,omitempty" bson:"onvif_credentials"`
Focus float64 `json:"focus,omitempty" bson:"focus"`
}
type OnvifPreset struct {
OnvifCredentials OnvifCredentials `json:"onvif_credentials,omitempty" bson:"onvif_credentials"`
Preset string `json:"preset,omitempty" bson:"preset"`
-1
View File
@@ -105,7 +105,6 @@ type IPCamera struct {
ONVIFXAddr string `json:"onvif_xaddr" bson:"onvif_xaddr"`
ONVIFUsername string `json:"onvif_username" bson:"onvif_username"`
ONVIFPassword string `json:"onvif_password" bson:"onvif_password"`
ONVIFChannel int `json:"onvif_channel,omitempty" bson:"onvif_channel,omitempty"` // 1-based video input on multi-channel devices (encoders, NVRs); 0 = first profile
SPSNALUs [][]byte `json:"sps_nalus,omitempty" bson:"sps_nalus,omitempty"`
PPSNALUs [][]byte `json:"pps_nalus,omitempty" bson:"pps_nalus,omitempty"`
VPSNALUs [][]byte `json:"vps_nalus,omitempty" bson:"vps_nalus,omitempty"`
-1
View File
@@ -12,7 +12,6 @@ type OnvifActionPTZ struct {
Down int `json:"down" bson:"down"`
Center int `json:"center" bson:"center"`
Zoom float64 `json:"zoom" bson:"zoom"`
Focus float64 `json:"focus" bson:"focus"`
X float64 `json:"x" bson:"x"`
Y float64 `json:"y" bson:"y"`
Z float64 `json:"z" bson:"z"`
@@ -11,15 +11,12 @@ const RecordingUploadMetadataExtension = ".metadata"
// with a recording. New optional fields can be added without changing the queue
// mechanism or breaking older agents.
type RecordingUploadMetadata struct {
FileName string `json:"filename"`
DeviceKey string `json:"device_key"`
Timestamp int64 `json:"timestamp"` // Unix milliseconds.
Duration uint64 `json:"duration"` // Milliseconds.
FPS float64 `json:"fps,omitempty"`
Encrypted bool `json:"encrypted,omitempty"`
DeviceName string `json:"device_name,omitempty"`
RegionCoordinates string `json:"region_coordinates,omitempty"`
NumberOfChanges string `json:"number_of_changes,omitempty"`
FileName string `json:"filename"`
DeviceKey string `json:"device_key"`
Timestamp int64 `json:"timestamp"` // Unix milliseconds.
Duration uint64 `json:"duration"` // Milliseconds.
FPS float64 `json:"fps,omitempty"`
Encrypted bool `json:"encrypted,omitempty"`
}
// RecordingUploadMetadataFileName returns the queue marker name associated
-113
View File
@@ -1,113 +0,0 @@
package onvif
import (
"errors"
"fmt"
"io"
"github.com/kerberos-io/onvif"
"github.com/kerberos-io/onvif/media"
xsdonvif "github.com/kerberos-io/onvif/xsd/onvif"
)
// Multi-channel devices (encoders, NVRs) expose several media profiles per
// video input, e.g. a Hanwha SPE-420 returns DefaultProfile-01-0..03-0 for
// channel 1 and DefaultProfile-01-1..03-1 for channel 2. PTZ, presets and
// focus must be addressed with a profile (or video source) of the channel the
// camera is wired to, otherwise commands drive the wrong camera.
//
// A channel is 1-based, counted in the order the device first lists each video
// source in GetProfiles. Channel 0 (unset) keeps the historic behaviour of
// using the first profile, so single-sensor cameras need no configuration.
// GetTokenFromChannel returns the media profile token to use for PTZ commands
// on the given 1-based channel.
func GetTokenFromChannel(device *onvif.Device, channel int) (xsdonvif.ReferenceToken, error) {
if channel <= 0 {
return GetTokenFromProfile(device, 0)
}
profiles, err := getMediaProfiles(device)
if err != nil {
return "", err
}
profile, err := selectChannelProfile(profiles, channel)
if err != nil {
return "", err
}
return profile.Token, nil
}
// GetVideoSourceTokenFromChannel returns the video source token of the given
// 1-based channel, as used by the imaging (focus) service.
func GetVideoSourceTokenFromChannel(device *onvif.Device, channel int) (xsdonvif.ReferenceToken, error) {
if channel <= 0 {
return GetVideoSourceTokenFromProfile(device, 0)
}
profiles, err := getMediaProfiles(device)
if err != nil {
return "", err
}
profile, err := selectChannelProfile(profiles, channel)
if err != nil {
return "", err
}
return *profile.VideoSourceConfiguration.SourceToken, nil
}
func getMediaProfiles(device *onvif.Device) ([]xsdonvif.Profile, error) {
resp, err := device.CallMethod(media.GetProfiles{})
if err != nil {
if resp != nil {
resp.Body.Close()
}
return nil, err
}
b, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return nil, err
}
decodedXML, et, err := getXMLNode(string(b), "GetProfilesResponse")
if err != nil {
return nil, err
}
var profilesResp media.GetProfilesResponse
if err := decodedXML.DecodeElement(&profilesResp, et); err != nil {
return nil, err
}
return profilesResp.Profiles, nil
}
// selectChannelProfile groups profiles by video source and returns a profile
// of the channel-th source (1-based). Within a channel it prefers the first
// profile that has a PTZ configuration attached.
func selectChannelProfile(profiles []xsdonvif.Profile, channel int) (xsdonvif.Profile, error) {
if channel <= 0 {
return xsdonvif.Profile{}, errors.New("onvif channel must be 1 or higher")
}
var sources []xsdonvif.ReferenceToken
bySource := map[xsdonvif.ReferenceToken][]xsdonvif.Profile{}
for _, profile := range profiles {
if profile.VideoSourceConfiguration == nil || profile.VideoSourceConfiguration.SourceToken == nil {
continue
}
source := *profile.VideoSourceConfiguration.SourceToken
if _, seen := bySource[source]; !seen {
sources = append(sources, source)
}
bySource[source] = append(bySource[source], profile)
}
if channel > len(sources) {
return xsdonvif.Profile{}, fmt.Errorf("onvif channel %d not found: device exposes %d video source(s)", channel, len(sources))
}
candidates := bySource[sources[channel-1]]
for _, profile := range candidates {
if profile.PTZConfiguration != nil {
return profile, nil
}
}
return candidates[0], nil
}
-105
View File
@@ -1,105 +0,0 @@
package onvif
import (
"fmt"
"testing"
xsdonvif "github.com/kerberos-io/onvif/xsd/onvif"
)
func testProfile(token string, source string, ptz bool) xsdonvif.Profile {
profile := xsdonvif.Profile{Token: xsdonvif.ReferenceToken(token)}
if source != "" {
sourceToken := xsdonvif.ReferenceToken(source)
profile.VideoSourceConfiguration = &xsdonvif.VideoSourceConfiguration{SourceToken: &sourceToken}
}
if ptz {
profile.PTZConfiguration = &xsdonvif.PTZConfiguration{}
}
return profile
}
// spe420Profiles mirrors a Hanwha SPE-420: three profiles per channel, token
// suffix -(N-1), listed grouped by profile type rather than by channel.
func spe420Profiles() []xsdonvif.Profile {
var profiles []xsdonvif.Profile
for kind := 1; kind <= 3; kind++ {
for ch := 0; ch < 4; ch++ {
profiles = append(profiles, testProfile(
fmt.Sprintf("DefaultProfile-0%d-%d", kind, ch),
fmt.Sprintf("VideoSourceToken-%d", ch),
true,
))
}
}
return profiles
}
func TestSelectChannelProfileMapsEncoderChannels(t *testing.T) {
profiles := spe420Profiles()
for channel := 1; channel <= 4; channel++ {
profile, err := selectChannelProfile(profiles, channel)
if err != nil {
t.Fatalf("channel %d: %v", channel, err)
}
want := fmt.Sprintf("DefaultProfile-01-%d", channel-1)
if string(profile.Token) != want {
t.Fatalf("channel %d: got %s, want %s", channel, profile.Token, want)
}
}
}
func TestSelectChannelProfilePrefersProfileWithPTZ(t *testing.T) {
profiles := []xsdonvif.Profile{
testProfile("mjpeg", "src-0", false),
testProfile("main", "src-0", true),
}
profile, err := selectChannelProfile(profiles, 1)
if err != nil {
t.Fatal(err)
}
if profile.Token != "main" {
t.Fatalf("got %s, want main", profile.Token)
}
}
func TestSelectChannelProfileFallsBackToFirstProfileOfChannel(t *testing.T) {
profiles := []xsdonvif.Profile{
testProfile("a", "src-0", false),
testProfile("b", "src-1", false),
testProfile("c", "src-1", false),
}
profile, err := selectChannelProfile(profiles, 2)
if err != nil {
t.Fatal(err)
}
if profile.Token != "b" {
t.Fatalf("got %s, want b", profile.Token)
}
}
func TestSelectChannelProfileSkipsProfilesWithoutVideoSource(t *testing.T) {
profiles := []xsdonvif.Profile{
testProfile("audio-only", "", false),
testProfile("video", "src-0", false),
}
profile, err := selectChannelProfile(profiles, 1)
if err != nil {
t.Fatal(err)
}
if profile.Token != "video" {
t.Fatalf("got %s, want video", profile.Token)
}
}
func TestSelectChannelProfileRejectsUnknownChannel(t *testing.T) {
if _, err := selectChannelProfile(spe420Profiles(), 5); err == nil {
t.Fatal("expected error for channel beyond the device's video sources")
}
if _, err := selectChannelProfile(spe420Profiles(), 0); err == nil {
t.Fatal("expected error for channel 0")
}
if _, err := selectChannelProfile(nil, 1); err == nil {
t.Fatal("expected error when the device has no profiles")
}
}
-466
View File
@@ -1,466 +0,0 @@
package onvif
import (
"encoding/xml"
"errors"
"fmt"
"io"
"math"
"net/http"
"regexp"
"slices"
"time"
"github.com/kerberos-io/onvif"
"github.com/kerberos-io/onvif/imaging"
"github.com/kerberos-io/onvif/media"
xsdonvif "github.com/kerberos-io/onvif/xsd/onvif"
log "github.com/sirupsen/logrus"
)
// Focus control goes through the ONVIF imaging service. The request types of
// the onvif library serialise every field (all three move types, every imaging
// setting), which cameras reject or which would overwrite unrelated imaging
// settings. The types below only carry the fields we want to send. Schema
// elements use the "onvif" prefix, which the library declares on the envelope
// as http://www.onvif.org/ver10/schema.
const (
focusModeAuto = "AUTO"
focusModeManual = "MANUAL"
// Same move duration as ContinuousZoom.
continuousFocusDuration = 500 * time.Millisecond
// Step for cameras that only support relative focus moves.
relativeFocusDistance = 0.1
// Step at full speed for cameras that only support absolute focus moves
// (e.g. Axis); scaled by the requested speed.
absoluteFocusStep = 0.1
// Time to let the camera settle in MANUAL before re-enabling AUTO.
autoFocusRetriggerDelay = 300 * time.Millisecond
)
type imagingMoveRequest struct {
XMLName xml.Name `xml:"timg:Move"`
VideoSourceToken xsdonvif.ReferenceToken `xml:"timg:VideoSourceToken"`
Focus focusMove `xml:"timg:Focus"`
}
type focusMove struct {
Absolute *absoluteFocusMove `xml:"onvif:Absolute,omitempty"`
Relative *relativeFocusMove `xml:"onvif:Relative,omitempty"`
Continuous *continuousFocusMove `xml:"onvif:Continuous,omitempty"`
}
type absoluteFocusMove struct {
Position float64 `xml:"onvif:Position"`
}
type relativeFocusMove struct {
Distance float64 `xml:"onvif:Distance"`
Speed float64 `xml:"onvif:Speed,omitempty"`
}
type continuousFocusMove struct {
Speed float64 `xml:"onvif:Speed"`
}
type imagingStopRequest struct {
XMLName xml.Name `xml:"timg:Stop"`
VideoSourceToken xsdonvif.ReferenceToken `xml:"timg:VideoSourceToken"`
}
type imagingGetSettingsRequest struct {
XMLName xml.Name `xml:"timg:GetImagingSettings"`
VideoSourceToken xsdonvif.ReferenceToken `xml:"timg:VideoSourceToken"`
}
type imagingGetSettingsResponse struct {
ImagingSettings struct {
Brightness *float64 `xml:"Brightness"`
Focus struct {
AutoFocusMode string `xml:"AutoFocusMode"`
} `xml:"Focus"`
} `xml:"ImagingSettings"`
}
type imagingSetFocusModeRequest struct {
XMLName xml.Name `xml:"timg:SetImagingSettings"`
VideoSourceToken xsdonvif.ReferenceToken `xml:"timg:VideoSourceToken"`
ImagingSettings struct {
Brightness *float64 `xml:"onvif:Brightness,omitempty"`
Focus struct {
AutoFocusMode string `xml:"onvif:AutoFocusMode"`
} `xml:"onvif:Focus"`
} `xml:"timg:ImagingSettings"`
}
type imagingGetOptionsRequest struct {
XMLName xml.Name `xml:"timg:GetOptions"`
VideoSourceToken xsdonvif.ReferenceToken `xml:"timg:VideoSourceToken"`
}
type imagingGetOptionsResponse struct {
ImagingOptions struct {
Focus struct {
AutoFocusModes []string `xml:"AutoFocusModes"`
} `xml:"Focus"`
} `xml:"ImagingOptions"`
}
type imagingGetMoveOptionsRequest struct {
XMLName xml.Name `xml:"timg:GetMoveOptions"`
VideoSourceToken xsdonvif.ReferenceToken `xml:"timg:VideoSourceToken"`
}
type imagingGetMoveOptionsResponse struct {
MoveOptions struct {
Absolute *struct {
Position struct {
Min float64 `xml:"Min"`
Max float64 `xml:"Max"`
} `xml:"Position"`
} `xml:"Absolute"`
Relative *struct{} `xml:"Relative"`
Continuous *struct{} `xml:"Continuous"`
} `xml:"MoveOptions"`
}
type imagingGetStatusRequest struct {
XMLName xml.Name `xml:"timg:GetStatus"`
VideoSourceToken xsdonvif.ReferenceToken `xml:"timg:VideoSourceToken"`
}
type imagingGetStatusResponse struct {
Status struct {
FocusStatus20 *struct {
Position *float64 `xml:"Position"`
} `xml:"FocusStatus20"`
FocusStatus *struct {
Position *float64 `xml:"Position"`
} `xml:"FocusStatus"`
} `xml:"Status"`
}
// focusMoveOptions lists the focus move types a camera supports.
type focusMoveOptions struct {
Continuous bool
Relative bool
Absolute bool
AbsoluteMin float64
AbsoluteMax float64
}
// allFocusMoveOptions is used when a camera does not report its move options:
// every move type is tried in turn.
var allFocusMoveOptions = focusMoveOptions{Continuous: true, Relative: true, Absolute: true, AbsoluteMin: 0, AbsoluteMax: 1}
// focusController abstracts the imaging calls so the focus logic can be tested
// without a camera.
type focusController interface {
FocusMode() (string, error)
FocusModes() ([]string, error)
SetFocusMode(mode string) error
MoveOptions() (focusMoveOptions, error)
FocusPosition() (float64, error)
MoveFocus(move focusMove) error
StopFocus() error
}
type onvifFocusClient struct {
device *onvif.Device
endpoint string
token xsdonvif.ReferenceToken
}
func newONVIFFocusClient(device *onvif.Device, channel int) (*onvifFocusClient, error) {
endpoint, err := device.GetEndpointByRequestStruct(&imaging.Move{})
if err != nil {
return nil, errors.New("camera does not expose the ONVIF imaging service: " + err.Error())
}
token, err := GetVideoSourceTokenFromChannel(device, channel)
if err != nil {
return nil, err
}
return &onvifFocusClient{device: device, endpoint: endpoint, token: token}, nil
}
var soapFaultReason = regexp.MustCompile(`<(?:[\w.-]+:)?Text[^>]*>([^<]+)</`)
// call sends an imaging request and returns the response body. A rejected
// request returns an error carrying the camera's SOAP fault reason.
func (client *onvifFocusClient) call(operation string, request interface{}) ([]byte, error) {
body, err := xml.Marshal(request)
if err != nil {
return nil, err
}
resp, err := client.device.SendSoap(client.endpoint, string(body))
if resp == nil {
if err == nil {
err = errors.New("no response")
}
return nil, fmt.Errorf("%s: %w", operation, err)
}
b, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
reason := http.StatusText(resp.StatusCode)
if m := soapFaultReason.FindSubmatch(b); m != nil {
reason = string(m[1])
}
return nil, fmt.Errorf("%s: camera returned HTTP %d: %s", operation, resp.StatusCode, reason)
}
if readErr != nil {
return nil, fmt.Errorf("%s: %w", operation, readErr)
}
return b, nil
}
func (client *onvifFocusClient) query(operation string, request interface{}, responseNode string, response interface{}) error {
b, err := client.call(operation, request)
if err != nil {
return err
}
decodedXML, et, err := getXMLNode(string(b), responseNode)
if err != nil {
return fmt.Errorf("%s: %w", operation, err)
}
if err := decodedXML.DecodeElement(response, et); err != nil {
return fmt.Errorf("%s: %w", operation, err)
}
return nil
}
func (client *onvifFocusClient) imagingSettings() (imagingGetSettingsResponse, error) {
var settings imagingGetSettingsResponse
err := client.query("get_imaging_settings", imagingGetSettingsRequest{VideoSourceToken: client.token}, "GetImagingSettingsResponse", &settings)
return settings, err
}
func (client *onvifFocusClient) FocusMode() (string, error) {
settings, err := client.imagingSettings()
return settings.ImagingSettings.Focus.AutoFocusMode, err
}
func (client *onvifFocusClient) SetFocusMode(mode string) error {
request := imagingSetFocusModeRequest{VideoSourceToken: client.token}
request.ImagingSettings.Focus.AutoFocusMode = mode
_, err := client.call("set_focus_mode", request)
if err == nil {
return nil
}
// Some cameras (seen on Axis) fail on a SetImagingSettings that only
// carries Focus. Retry with the current brightness, which leaves it as is.
settings, settingsErr := client.imagingSettings()
if settingsErr != nil || settings.ImagingSettings.Brightness == nil {
return err
}
request.ImagingSettings.Brightness = settings.ImagingSettings.Brightness
if _, retryErr := client.call("set_focus_mode", request); retryErr != nil {
return errors.Join(err, retryErr)
}
return nil
}
func (client *onvifFocusClient) FocusModes() ([]string, error) {
var response imagingGetOptionsResponse
if err := client.query("get_imaging_options", imagingGetOptionsRequest{VideoSourceToken: client.token}, "GetOptionsResponse", &response); err != nil {
return nil, err
}
return response.ImagingOptions.Focus.AutoFocusModes, nil
}
func (client *onvifFocusClient) MoveOptions() (focusMoveOptions, error) {
var response imagingGetMoveOptionsResponse
if err := client.query("get_focus_move_options", imagingGetMoveOptionsRequest{VideoSourceToken: client.token}, "GetMoveOptionsResponse", &response); err != nil {
return focusMoveOptions{}, err
}
moveOptions := response.MoveOptions
options := focusMoveOptions{
Continuous: moveOptions.Continuous != nil,
Relative: moveOptions.Relative != nil,
Absolute: moveOptions.Absolute != nil,
AbsoluteMin: 0,
AbsoluteMax: 1,
}
if moveOptions.Absolute != nil && moveOptions.Absolute.Position.Max > moveOptions.Absolute.Position.Min {
options.AbsoluteMin = moveOptions.Absolute.Position.Min
options.AbsoluteMax = moveOptions.Absolute.Position.Max
}
return options, nil
}
func (client *onvifFocusClient) FocusPosition() (float64, error) {
var response imagingGetStatusResponse
if err := client.query("get_focus_status", imagingGetStatusRequest{VideoSourceToken: client.token}, "GetStatusResponse", &response); err != nil {
return 0, err
}
status := response.Status
if status.FocusStatus20 != nil && status.FocusStatus20.Position != nil {
return *status.FocusStatus20.Position, nil
}
if status.FocusStatus != nil && status.FocusStatus.Position != nil {
return *status.FocusStatus.Position, nil
}
return 0, errors.New("get_focus_status: camera did not report a focus position")
}
func (client *onvifFocusClient) MoveFocus(move focusMove) error {
_, err := client.call("focus_move", imagingMoveRequest{VideoSourceToken: client.token, Focus: move})
return err
}
func (client *onvifFocusClient) StopFocus() error {
_, err := client.call("focus_stop", imagingStopRequest{VideoSourceToken: client.token})
return err
}
// ContinuousFocus moves the focus motor a short step, the focus equivalent of
// ContinuousZoom. Speed is in [-1, 1]: positive focuses near, negative focuses
// far. Cameras with continuous focus move for a moment and stop; cameras with
// only relative or absolute focus (e.g. Axis) step by an amount scaled to the
// speed. The camera is switched to MANUAL focus first, because cameras ignore
// manual moves while autofocus is active; use AutoFocus to hand control back.
func ContinuousFocus(device *onvif.Device, channel int, speed float64) error {
client, err := newONVIFFocusClient(device, channel)
if err != nil {
return err
}
return continuousFocus(client, speed, continuousFocusDuration)
}
// AutoFocus triggers the camera's native ONVIF autofocus. Setting AUTO while
// already in AUTO is a no-op on most cameras, so we toggle through MANUAL to
// force a fresh focus cycle.
func AutoFocus(device *onvif.Device, channel int) error {
client, err := newONVIFFocusClient(device, channel)
if err != nil {
return err
}
return autoFocus(client, autoFocusRetriggerDelay)
}
func continuousFocus(client focusController, speed float64, duration time.Duration) error {
speed = math.Max(-1, math.Min(1, speed))
if speed == 0 {
return client.StopFocus()
}
if err := ensureFocusMode(client, focusModeManual); err != nil {
// Some cameras have no focus mode setting but still accept moves.
log.Debug("onvif.continuousFocus(): could not switch to manual focus: " + err.Error())
}
options, err := client.MoveOptions()
if err != nil {
log.Debug("onvif.continuousFocus(): could not read focus move options, trying all move types: " + err.Error())
options = allFocusMoveOptions
}
var moveErrs []error
if options.Continuous {
err := client.MoveFocus(focusMove{Continuous: &continuousFocusMove{Speed: speed}})
if err == nil {
time.Sleep(duration)
return client.StopFocus()
}
moveErrs = append(moveErrs, err)
}
if options.Relative {
// A single relative step needs no stop.
err := client.MoveFocus(focusMove{Relative: &relativeFocusMove{
Distance: math.Copysign(relativeFocusDistance, speed),
Speed: math.Abs(speed),
}})
if err == nil {
return nil
}
moveErrs = append(moveErrs, err)
}
if options.Absolute {
position, err := client.FocusPosition()
if err == nil {
target := math.Max(options.AbsoluteMin, math.Min(options.AbsoluteMax, position+speed*absoluteFocusStep))
err = client.MoveFocus(focusMove{Absolute: &absoluteFocusMove{Position: target}})
}
if err == nil {
return nil
}
moveErrs = append(moveErrs, err)
}
if len(moveErrs) == 0 {
return errors.New("camera does not support any focus move")
}
return errors.Join(moveErrs...)
}
func ensureFocusMode(client focusController, mode string) error {
current, err := client.FocusMode()
if err == nil && current == mode {
return nil
}
return client.SetFocusMode(mode)
}
var errNoNativeAutoFocus = errors.New("camera does not support native ONVIF autofocus")
func autoFocus(client focusController, delay time.Duration) error {
// Some cameras (seen on Dahua) only advertise MANUAL, yet accept a switch
// to AUTO and silently ignore it.
if modes, err := client.FocusModes(); err == nil && !slices.Contains(modes, focusModeAuto) {
return errNoNativeAutoFocus
}
mode, err := client.FocusMode()
if err == nil && mode == focusModeAuto {
if err := client.SetFocusMode(focusModeManual); err != nil {
return err
}
time.Sleep(delay)
}
if err := client.SetFocusMode(focusModeAuto); err != nil {
return err
}
if mode, err := client.FocusMode(); err == nil && mode != focusModeAuto {
return fmt.Errorf("%w: camera accepted AUTO but reports %s", errNoNativeAutoFocus, mode)
}
return nil
}
// GetVideoSourceTokenFromProfile returns the video source token of a media
// profile; the imaging service is addressed by video source, not by profile.
func GetVideoSourceTokenFromProfile(device *onvif.Device, profileId int) (xsdonvif.ReferenceToken, error) {
var sourceToken xsdonvif.ReferenceToken
resp, err := device.CallMethod(media.GetProfiles{})
if err != nil {
if resp != nil {
resp.Body.Close()
}
return sourceToken, err
}
b, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return sourceToken, err
}
decodedXML, et, err := getXMLNode(string(b), "GetProfilesResponse")
if err != nil {
return sourceToken, err
}
var mProfilesResp media.GetProfilesResponse
if err := decodedXML.DecodeElement(&mProfilesResp, et); err != nil {
return sourceToken, err
}
for i, profile := range mProfilesResp.Profiles {
if i == profileId && profile.VideoSourceConfiguration != nil && profile.VideoSourceConfiguration.SourceToken != nil {
return *profile.VideoSourceConfiguration.SourceToken, nil
}
}
return sourceToken, errors.New("no video source token found for profile")
}
-418
View File
@@ -1,418 +0,0 @@
package onvif
import (
"encoding/xml"
"errors"
"math"
"strings"
"testing"
)
type fakeFocusClient struct {
mode string
modeErr error
focusModes []string
focusModesErr error
ignoreAuto bool
setModeErr error
options *focusMoveOptions
optionsErr error
position float64
positionErr error
continuousErr error
relativeErr error
absoluteErr error
modes []string
moves []focusMove
stopCalls int
}
func (client *fakeFocusClient) FocusMode() (string, error) {
return client.mode, client.modeErr
}
func (client *fakeFocusClient) FocusModes() ([]string, error) {
if client.focusModes == nil && client.focusModesErr == nil {
return []string{focusModeManual, focusModeAuto}, nil
}
return client.focusModes, client.focusModesErr
}
func (client *fakeFocusClient) SetFocusMode(mode string) error {
client.modes = append(client.modes, mode)
if client.setModeErr != nil {
return client.setModeErr
}
if mode == focusModeAuto && client.ignoreAuto {
return nil
}
client.mode = mode
return nil
}
func (client *fakeFocusClient) MoveOptions() (focusMoveOptions, error) {
if client.optionsErr != nil {
return focusMoveOptions{}, client.optionsErr
}
if client.options == nil {
return allFocusMoveOptions, nil
}
return *client.options, nil
}
func (client *fakeFocusClient) FocusPosition() (float64, error) {
return client.position, client.positionErr
}
func (client *fakeFocusClient) MoveFocus(move focusMove) error {
client.moves = append(client.moves, move)
switch {
case move.Continuous != nil:
return client.continuousErr
case move.Relative != nil:
return client.relativeErr
default:
return client.absoluteErr
}
}
func (client *fakeFocusClient) StopFocus() error {
client.stopCalls++
return nil
}
func TestContinuousFocusSwitchesToManualMovesAndStops(t *testing.T) {
client := &fakeFocusClient{mode: focusModeAuto}
if err := continuousFocus(client, 0.5, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if len(client.modes) != 1 || client.modes[0] != focusModeManual {
t.Fatalf("modes = %v, want [MANUAL]", client.modes)
}
if len(client.moves) != 1 || client.moves[0].Continuous == nil || client.moves[0].Continuous.Speed != 0.5 {
t.Fatalf("moves = %+v, want one continuous move at 0.5", client.moves)
}
if client.stopCalls != 1 {
t.Fatalf("stopCalls = %d, want 1", client.stopCalls)
}
}
func TestContinuousFocusKeepsManualMode(t *testing.T) {
client := &fakeFocusClient{mode: focusModeManual}
if err := continuousFocus(client, 0.5, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if len(client.modes) != 0 {
t.Fatalf("modes = %v, want no mode change when already MANUAL", client.modes)
}
}
func TestContinuousFocusClampsSpeed(t *testing.T) {
client := &fakeFocusClient{}
if err := continuousFocus(client, -3, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if got := client.moves[0].Continuous.Speed; got != -1 {
t.Fatalf("speed = %v, want -1", got)
}
}
func TestContinuousFocusIgnoresUnsupportedFocusMode(t *testing.T) {
client := &fakeFocusClient{setModeErr: errors.New("not supported")}
if err := continuousFocus(client, 0.5, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if len(client.moves) != 1 || client.stopCalls != 1 {
t.Fatalf("moves = %d stopCalls = %d, want 1 and 1", len(client.moves), client.stopCalls)
}
}
func TestContinuousFocusFallsBackToRelative(t *testing.T) {
client := &fakeFocusClient{continuousErr: errors.New("continuous not supported")}
if err := continuousFocus(client, -0.4, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if len(client.moves) != 2 || client.moves[1].Relative == nil {
t.Fatalf("moves = %+v, want continuous then relative", client.moves)
}
relative := client.moves[1].Relative
if relative.Distance != -relativeFocusDistance || relative.Speed != 0.4 {
t.Fatalf("relative = %+v, want distance %v speed 0.4", relative, -relativeFocusDistance)
}
if client.stopCalls != 0 {
t.Fatalf("stopCalls = %d, want 0 for a relative move", client.stopCalls)
}
}
func TestContinuousFocusUsesAbsoluteWhenOnlyAbsoluteSupported(t *testing.T) {
// Axis cameras report only absolute focus moves.
client := &fakeFocusClient{
mode: focusModeManual,
options: &focusMoveOptions{Absolute: true, AbsoluteMin: 0, AbsoluteMax: 1},
position: 0.48,
}
if err := continuousFocus(client, 0.5, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if len(client.moves) != 1 || client.moves[0].Absolute == nil {
t.Fatalf("moves = %+v, want a single absolute move", client.moves)
}
if got, want := client.moves[0].Absolute.Position, 0.48+0.5*absoluteFocusStep; math.Abs(got-want) > 1e-9 {
t.Fatalf("position = %v, want %v", got, want)
}
if client.stopCalls != 0 {
t.Fatalf("stopCalls = %d, want 0 for an absolute move", client.stopCalls)
}
}
func TestContinuousFocusClampsAbsoluteToRange(t *testing.T) {
client := &fakeFocusClient{
options: &focusMoveOptions{Absolute: true, AbsoluteMin: 0, AbsoluteMax: 1},
position: 0.02,
}
if err := continuousFocus(client, -1, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if got := client.moves[0].Absolute.Position; got != 0 {
t.Fatalf("position = %v, want clamped to 0", got)
}
}
func TestContinuousFocusTriesAllMovesWithoutOptions(t *testing.T) {
client := &fakeFocusClient{
optionsErr: errors.New("no move options"),
continuousErr: errors.New("continuous not supported"),
relativeErr: errors.New("relative not supported"),
position: 0.5,
}
if err := continuousFocus(client, 0.5, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if len(client.moves) != 3 || client.moves[2].Absolute == nil {
t.Fatalf("moves = %+v, want continuous, relative then absolute", client.moves)
}
}
func TestContinuousFocusReturnsErrorWhenAllMovesFail(t *testing.T) {
client := &fakeFocusClient{
continuousErr: errors.New("continuous not supported"),
relativeErr: errors.New("relative not supported"),
positionErr: errors.New("no position"),
}
err := continuousFocus(client, 0.5, 0)
if err == nil || !strings.Contains(err.Error(), "continuous") || !strings.Contains(err.Error(), "relative") || !strings.Contains(err.Error(), "position") {
t.Fatalf("continuousFocus() error = %v, want every move error", err)
}
}
func TestContinuousFocusReturnsErrorWhenNoMoveSupported(t *testing.T) {
client := &fakeFocusClient{options: &focusMoveOptions{}}
if err := continuousFocus(client, 0.5, 0); err == nil {
t.Fatal("continuousFocus() error = nil, want an error")
}
if len(client.moves) != 0 {
t.Fatalf("moves = %+v, want none", client.moves)
}
}
func TestContinuousFocusZeroSpeedStops(t *testing.T) {
client := &fakeFocusClient{}
if err := continuousFocus(client, 0, 0); err != nil {
t.Fatalf("continuousFocus() error = %v", err)
}
if len(client.moves) != 0 || len(client.modes) != 0 || client.stopCalls != 1 {
t.Fatalf("moves = %d modes = %v stopCalls = %d, want only a stop", len(client.moves), client.modes, client.stopCalls)
}
}
func TestAutoFocusRetriggersWhenAlreadyAuto(t *testing.T) {
client := &fakeFocusClient{mode: focusModeAuto}
if err := autoFocus(client, 0); err != nil {
t.Fatalf("autoFocus() error = %v", err)
}
if strings.Join(client.modes, ",") != "MANUAL,AUTO" {
t.Fatalf("modes = %v, want [MANUAL AUTO]", client.modes)
}
}
func TestAutoFocusFromManual(t *testing.T) {
client := &fakeFocusClient{mode: focusModeManual}
if err := autoFocus(client, 0); err != nil {
t.Fatalf("autoFocus() error = %v", err)
}
if strings.Join(client.modes, ",") != "AUTO" {
t.Fatalf("modes = %v, want [AUTO]", client.modes)
}
}
func TestAutoFocusWhenModeUnreadable(t *testing.T) {
client := &fakeFocusClient{modeErr: errors.New("unreadable")}
if err := autoFocus(client, 0); err != nil {
t.Fatalf("autoFocus() error = %v", err)
}
if strings.Join(client.modes, ",") != "AUTO" {
t.Fatalf("modes = %v, want [AUTO]", client.modes)
}
}
func TestAutoFocusRejectsCameraWithoutAutoMode(t *testing.T) {
// Dahua cameras advertise only MANUAL.
client := &fakeFocusClient{mode: focusModeManual, focusModes: []string{focusModeManual}}
if err := autoFocus(client, 0); !errors.Is(err, errNoNativeAutoFocus) {
t.Fatalf("autoFocus() error = %v, want errNoNativeAutoFocus", err)
}
if len(client.modes) != 0 {
t.Fatalf("modes = %v, want no mode change", client.modes)
}
}
func TestAutoFocusDetectsIgnoredAutoMode(t *testing.T) {
client := &fakeFocusClient{mode: focusModeManual, ignoreAuto: true}
if err := autoFocus(client, 0); !errors.Is(err, errNoNativeAutoFocus) {
t.Fatalf("autoFocus() error = %v, want errNoNativeAutoFocus", err)
}
}
func TestAutoFocusWhenFocusModesUnreadable(t *testing.T) {
client := &fakeFocusClient{mode: focusModeManual, focusModesErr: errors.New("no options")}
if err := autoFocus(client, 0); err != nil {
t.Fatalf("autoFocus() error = %v", err)
}
if strings.Join(client.modes, ",") != "AUTO" {
t.Fatalf("modes = %v, want [AUTO]", client.modes)
}
}
func TestImagingRequestsOnlyCarryFocusFields(t *testing.T) {
move, err := xml.Marshal(imagingMoveRequest{
VideoSourceToken: "source",
Focus: focusMove{Continuous: &continuousFocusMove{Speed: 0.5}},
})
if err != nil {
t.Fatalf("marshal move: %v", err)
}
wantMove := "<timg:Move><timg:VideoSourceToken>source</timg:VideoSourceToken><timg:Focus><onvif:Continuous><onvif:Speed>0.5</onvif:Speed></onvif:Continuous></timg:Focus></timg:Move>"
if string(move) != wantMove {
t.Fatalf("move XML = %s\nwant %s", move, wantMove)
}
absolute, err := xml.Marshal(imagingMoveRequest{
VideoSourceToken: "source",
Focus: focusMove{Absolute: &absoluteFocusMove{Position: 0.53}},
})
if err != nil {
t.Fatalf("marshal absolute move: %v", err)
}
wantAbsolute := "<timg:Move><timg:VideoSourceToken>source</timg:VideoSourceToken><timg:Focus><onvif:Absolute><onvif:Position>0.53</onvif:Position></onvif:Absolute></timg:Focus></timg:Move>"
if string(absolute) != wantAbsolute {
t.Fatalf("absolute move XML = %s\nwant %s", absolute, wantAbsolute)
}
request := imagingSetFocusModeRequest{VideoSourceToken: "source"}
request.ImagingSettings.Focus.AutoFocusMode = focusModeManual
settings, err := xml.Marshal(request)
if err != nil {
t.Fatalf("marshal settings: %v", err)
}
wantSettings := "<timg:SetImagingSettings><timg:VideoSourceToken>source</timg:VideoSourceToken><timg:ImagingSettings><onvif:Focus><onvif:AutoFocusMode>MANUAL</onvif:AutoFocusMode></onvif:Focus></timg:ImagingSettings></timg:SetImagingSettings>"
if string(settings) != wantSettings {
t.Fatalf("settings XML = %s\nwant %s", settings, wantSettings)
}
// Brightness must come before Focus (schema order) for the Axis retry.
brightness := 50.0
request.ImagingSettings.Brightness = &brightness
settings, err = xml.Marshal(request)
if err != nil {
t.Fatalf("marshal settings with brightness: %v", err)
}
wantSettings = "<timg:SetImagingSettings><timg:VideoSourceToken>source</timg:VideoSourceToken><timg:ImagingSettings><onvif:Brightness>50</onvif:Brightness><onvif:Focus><onvif:AutoFocusMode>MANUAL</onvif:AutoFocusMode></onvif:Focus></timg:ImagingSettings></timg:SetImagingSettings>"
if string(settings) != wantSettings {
t.Fatalf("settings XML = %s\nwant %s", settings, wantSettings)
}
}
const testImagingEnvelope = `<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://www.w3.org/2003/05/soap-envelope" xmlns:timg="http://www.onvif.org/ver20/imaging/wsdl" xmlns:tt="http://www.onvif.org/ver10/schema"><SOAP-ENV:Body>%s</SOAP-ENV:Body></SOAP-ENV:Envelope>`
func decodeTestImagingResponse(t *testing.T, body string, node string, out interface{}) {
t.Helper()
decoder, start, err := getXMLNode(strings.Replace(testImagingEnvelope, "%s", body, 1), node)
if err != nil {
t.Fatalf("getXMLNode() error = %v", err)
}
if err := decoder.DecodeElement(out, start); err != nil {
t.Fatalf("DecodeElement() error = %v", err)
}
}
func TestImagingSettingsResponseDecodesFocusModeAndBrightness(t *testing.T) {
var settings imagingGetSettingsResponse
decodeTestImagingResponse(t, `<timg:GetImagingSettingsResponse><timg:ImagingSettings><tt:Brightness>50</tt:Brightness><tt:Focus><tt:AutoFocusMode>AUTO</tt:AutoFocusMode></tt:Focus></timg:ImagingSettings></timg:GetImagingSettingsResponse>`, "GetImagingSettingsResponse", &settings)
if settings.ImagingSettings.Focus.AutoFocusMode != focusModeAuto {
t.Fatalf("AutoFocusMode = %q, want AUTO", settings.ImagingSettings.Focus.AutoFocusMode)
}
if settings.ImagingSettings.Brightness == nil || *settings.ImagingSettings.Brightness != 50 {
t.Fatalf("Brightness = %v, want 50", settings.ImagingSettings.Brightness)
}
}
func TestImagingOptionsResponseDecodesFocusModes(t *testing.T) {
// Response captured from a Dahua camera.
var response imagingGetOptionsResponse
decodeTestImagingResponse(t, `<timg:GetOptionsResponse><timg:ImagingOptions><tt:Brightness><tt:Min>0</tt:Min><tt:Max>100</tt:Max></tt:Brightness><tt:Focus><tt:AutoFocusModes>MANUAL</tt:AutoFocusModes><tt:DefaultSpeed><tt:Min>-1</tt:Min><tt:Max>1</tt:Max></tt:DefaultSpeed></tt:Focus></timg:ImagingOptions></timg:GetOptionsResponse>`, "GetOptionsResponse", &response)
if got := strings.Join(response.ImagingOptions.Focus.AutoFocusModes, ","); got != "MANUAL" {
t.Fatalf("AutoFocusModes = %q, want MANUAL", got)
}
}
func TestMoveOptionsResponseDecodesAbsoluteOnly(t *testing.T) {
// Response captured from an Axis camera.
var response imagingGetMoveOptionsResponse
decodeTestImagingResponse(t, `<timg:GetMoveOptionsResponse><timg:MoveOptions><tt:Absolute><tt:Position><tt:Min>0</tt:Min><tt:Max>1</tt:Max></tt:Position></tt:Absolute></timg:MoveOptions></timg:GetMoveOptionsResponse>`, "GetMoveOptionsResponse", &response)
options := response.MoveOptions
if options.Absolute == nil || options.Relative != nil || options.Continuous != nil {
t.Fatalf("options = %+v, want only absolute", options)
}
if options.Absolute.Position.Min != 0 || options.Absolute.Position.Max != 1 {
t.Fatalf("absolute range = %+v, want 0..1", options.Absolute.Position)
}
}
func TestStatusResponseDecodesFocusPosition(t *testing.T) {
// Response captured from an Axis camera.
var response imagingGetStatusResponse
decodeTestImagingResponse(t, `<timg:GetStatusResponse><timg:Status><tt:FocusStatus20><tt:Position>0.479226589</tt:Position><tt:MoveStatus>IDLE</tt:MoveStatus></tt:FocusStatus20></timg:Status></timg:GetStatusResponse>`, "GetStatusResponse", &response)
if response.Status.FocusStatus20 == nil || response.Status.FocusStatus20.Position == nil || *response.Status.FocusStatus20.Position != 0.479226589 {
t.Fatalf("status = %+v, want position 0.479226589", response.Status)
}
}
func TestSOAPFaultReasonExtraction(t *testing.T) {
fault := `<SOAP-ENV:Fault><SOAP-ENV:Reason><SOAP-ENV:Text xml:lang="en">Argument value invalid</SOAP-ENV:Text></SOAP-ENV:Reason></SOAP-ENV:Fault>`
m := soapFaultReason.FindStringSubmatch(fault)
if m == nil || m[1] != "Argument value invalid" {
t.Fatalf("fault reason = %v, want Argument value invalid", m)
}
}
+14 -35
View File
@@ -119,30 +119,10 @@ func HandleONVIFActions(configuration *models.Configuration, communication *mode
// Connect to Onvif device
cameraConfiguration := configuration.Config.Capture.IPCamera
device, _, err := ConnectToOnvifDevice(&cameraConfiguration)
if err == nil && (onvifAction.Action == "focus" || onvifAction.Action == "autofocus") {
if err == nil {
// Focus goes through the imaging service, so it also works on
// cameras without PTZ (e.g. fixed cameras with a motorised lens).
if onvifAction.Action == "focus" {
err := ContinuousFocus(device, cameraConfiguration.ONVIFChannel, ptzAction.Focus)
if err != nil {
log.Error("onvif.HandleONVIFActions() - ContinuousFocus: " + err.Error())
} else {
log.Info("onvif.HandleONVIFActions() - ContinuousFocus: successfully focused camera")
}
} else {
err := AutoFocus(device, cameraConfiguration.ONVIFChannel)
if err != nil {
log.Error("onvif.HandleONVIFActions() - AutoFocus: " + err.Error())
} else {
log.Info("onvif.HandleONVIFActions() - AutoFocus: successfully triggered autofocus")
}
}
} else if err == nil {
// Get the profile token of the configured channel
token, err := GetTokenFromChannel(device, cameraConfiguration.ONVIFChannel)
// Get token from the first profile
token, err := GetTokenFromProfile(device, 0)
if err == nil {
// Get the configurations from the device
@@ -174,7 +154,7 @@ func HandleONVIFActions(configuration *models.Configuration, communication *mode
// Execute the preset
preset := ptzAction.Preset
err := GoToPresetFromDevice(device, cameraConfiguration.ONVIFChannel, preset)
err := GoToPresetFromDevice(device, preset)
if err != nil {
log.Debug("onvif.HandleONVIFActions() - GotoPreset: " + err.Error())
} else {
@@ -369,8 +349,8 @@ func GetPositionFromDevice(configuration models.Configuration) (xsdonvif.PTZVect
device, _, err := ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
// Get the profile token of the configured channel
token, err := GetTokenFromChannel(device, cameraConfiguration.ONVIFChannel)
// Get token from the first profile
token, err := GetTokenFromProfile(device, 0)
if err == nil {
// Get the PTZ configurations from the device
position, err := GetPosition(device, token)
@@ -883,12 +863,12 @@ func GetCapabilitiesFromDevice(dev *onvif.Device) []string {
return capabilities
}
func GetPresetsFromDevice(device *onvif.Device, channel int) ([]models.OnvifActionPreset, error) {
func GetPresetsFromDevice(device *onvif.Device) ([]models.OnvifActionPreset, error) {
var presets []models.OnvifActionPreset
var presetsResponse ptz.GetPresetsResponse
// Get the profile token of the requested channel
token, err := GetTokenFromChannel(device, channel)
// Get token from the first profile
token, err := GetTokenFromProfile(device, 0)
if err == nil {
resp, err := device.CallMethod(ptz.GetPresets{
ProfileToken: token,
@@ -933,11 +913,11 @@ func GetPresetsFromDevice(device *onvif.Device, channel int) ([]models.OnvifActi
return presets, err
}
func GoToPresetFromDevice(device *onvif.Device, channel int, presetName string) error {
func GoToPresetFromDevice(device *onvif.Device, presetName string) error {
var goToPresetResponse ptz.GotoPresetResponse
// Get the profile token of the requested channel
token, err := GetTokenFromChannel(device, channel)
// Get token from the first profile
token, err := GetTokenFromProfile(device, 0)
if err == nil {
preset := xsdonvif.ReferenceToken(presetName)
resp, err := device.CallMethod(ptz.GotoPreset{
@@ -1040,7 +1020,6 @@ func VerifyOnvifConnection(c *gin.Context) {
ONVIFXAddr: onvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifCredentials.ONVIFUsername,
ONVIFPassword: onvifCredentials.ONVIFPassword,
ONVIFChannel: onvifCredentials.ONVIFChannel,
},
},
},
@@ -1049,8 +1028,8 @@ func VerifyOnvifConnection(c *gin.Context) {
cameraConfiguration := configuration.Config.Capture.IPCamera
device, capabilities, err := ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
// Get the profile token of the requested channel
token, err := GetTokenFromChannel(device, cameraConfiguration.ONVIFChannel)
// Get token from the first profile
token, err := GetTokenFromProfile(device, 0)
if err == nil {
c.JSON(200, gin.H{
"device": device,
+8 -66
View File
@@ -1,7 +1,6 @@
package http
import (
"errors"
"strconv"
"strings"
"time"
@@ -75,7 +74,6 @@ func LoginToOnvif(c *gin.Context) {
ONVIFXAddr: onvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifCredentials.ONVIFUsername,
ONVIFPassword: onvifCredentials.ONVIFPassword,
ONVIFChannel: onvifCredentials.ONVIFChannel,
},
},
},
@@ -84,8 +82,8 @@ func LoginToOnvif(c *gin.Context) {
cameraConfiguration := configuration.Config.Capture.IPCamera
device, capabilities, err := onvif.ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
// Get the profile token of the requested channel
token, err := onvif.GetTokenFromChannel(device, cameraConfiguration.ONVIFChannel)
// Get token from the first profile
token, err := onvif.GetTokenFromProfile(device, 0)
if err == nil {
c.JSON(200, gin.H{
"device": device,
@@ -130,7 +128,6 @@ func GetOnvifCapabilities(c *gin.Context) {
ONVIFXAddr: onvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifCredentials.ONVIFUsername,
ONVIFPassword: onvifCredentials.ONVIFPassword,
ONVIFChannel: onvifCredentials.ONVIFChannel,
},
},
},
@@ -175,7 +172,6 @@ func DoOnvifPanTilt(c *gin.Context) {
ONVIFXAddr: onvifPanTilt.OnvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifPanTilt.OnvifCredentials.ONVIFUsername,
ONVIFPassword: onvifPanTilt.OnvifCredentials.ONVIFPassword,
ONVIFChannel: onvifPanTilt.OnvifCredentials.ONVIFChannel,
},
},
},
@@ -185,8 +181,8 @@ func DoOnvifPanTilt(c *gin.Context) {
device, _, err := onvif.ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
// Get the profile token of the requested channel
token, err := onvif.GetTokenFromChannel(device, cameraConfiguration.ONVIFChannel)
// Get token from the first profile
token, err := onvif.GetTokenFromProfile(device, 0)
if err == nil {
@@ -250,7 +246,6 @@ func DoOnvifZoom(c *gin.Context) {
ONVIFXAddr: onvifZoom.OnvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifZoom.OnvifCredentials.ONVIFUsername,
ONVIFPassword: onvifZoom.OnvifCredentials.ONVIFPassword,
ONVIFChannel: onvifZoom.OnvifCredentials.ONVIFChannel,
},
},
},
@@ -260,8 +255,8 @@ func DoOnvifZoom(c *gin.Context) {
device, _, err := onvif.ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
// Get the profile token of the requested channel
token, err := onvif.GetTokenFromChannel(device, cameraConfiguration.ONVIFChannel)
// Get token from the first profile
token, err := onvif.GetTokenFromProfile(device, 0)
if err == nil {
@@ -303,54 +298,6 @@ func DoOnvifZoom(c *gin.Context) {
}
}
// DoOnvifFocus godoc
// @Router /api/camera/onvif/focus [post]
// @ID camera-onvif-focus
// @Tags onvif
// @Param focus body models.OnvifFocus true "OnvifFocus"
// @Summary Focusing the camera near or far.
// @Description Focusing the camera near (positive) or far (negative).
// @Success 200 {object} models.APIResponse
func DoOnvifFocus(c *gin.Context) {
var onvifFocus models.OnvifFocus
err := c.BindJSON(&onvifFocus)
if err == nil && onvifFocus.OnvifCredentials.ONVIFXAddr == "" {
err = errors.New("missing onvif_xaddr")
}
if err == nil {
cameraConfiguration := models.IPCamera{
ONVIFXAddr: onvifFocus.OnvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifFocus.OnvifCredentials.ONVIFUsername,
ONVIFPassword: onvifFocus.OnvifCredentials.ONVIFPassword,
ONVIFChannel: onvifFocus.OnvifCredentials.ONVIFChannel,
}
device, _, err := onvif.ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
err := onvif.ContinuousFocus(device, cameraConfiguration.ONVIFChannel, onvifFocus.Focus)
if err == nil {
c.JSON(200, models.APIResponse{
Message: "Successfully focused the camera",
})
} else {
c.JSON(400, models.APIResponse{
Message: "Something went wrong: " + err.Error(),
})
}
} else {
c.JSON(400, models.APIResponse{
Message: "Something went wrong: " + err.Error(),
})
}
} else {
c.JSON(400, models.APIResponse{
Message: "Something went wrong: " + err.Error(),
})
}
}
// GetOnvifPresets godoc
// @Router /api/camera/onvif/presets [post]
// @ID camera-onvif-presets
@@ -372,7 +319,6 @@ func GetOnvifPresets(c *gin.Context) {
ONVIFXAddr: onvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifCredentials.ONVIFUsername,
ONVIFPassword: onvifCredentials.ONVIFPassword,
ONVIFChannel: onvifCredentials.ONVIFChannel,
},
},
},
@@ -381,7 +327,7 @@ func GetOnvifPresets(c *gin.Context) {
cameraConfiguration := configuration.Config.Capture.IPCamera
device, _, err := onvif.ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
presets, err := onvif.GetPresetsFromDevice(device, cameraConfiguration.ONVIFChannel)
presets, err := onvif.GetPresetsFromDevice(device)
if err == nil {
c.JSON(200, gin.H{
"presets": presets,
@@ -424,7 +370,6 @@ func GoToOnvifPreset(c *gin.Context) {
ONVIFXAddr: onvifPreset.OnvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifPreset.OnvifCredentials.ONVIFUsername,
ONVIFPassword: onvifPreset.OnvifCredentials.ONVIFPassword,
ONVIFChannel: onvifPreset.OnvifCredentials.ONVIFChannel,
},
},
},
@@ -433,7 +378,7 @@ func GoToOnvifPreset(c *gin.Context) {
cameraConfiguration := configuration.Config.Capture.IPCamera
device, _, err := onvif.ConnectToOnvifDevice(&cameraConfiguration)
if err == nil {
err := onvif.GoToPresetFromDevice(device, cameraConfiguration.ONVIFChannel, onvifPreset.Preset)
err := onvif.GoToPresetFromDevice(device, onvifPreset.Preset)
if err == nil {
c.JSON(200, gin.H{
"data": "Camera preset activated: " + onvifPreset.Preset,
@@ -480,7 +425,6 @@ func DoGetDigitalInputs(c *gin.Context) {
ONVIFXAddr: onvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifCredentials.ONVIFUsername,
ONVIFPassword: onvifCredentials.ONVIFPassword,
ONVIFChannel: onvifCredentials.ONVIFChannel,
},
},
},
@@ -570,7 +514,6 @@ func DoGetRelayOutputs(c *gin.Context) {
ONVIFXAddr: onvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifCredentials.ONVIFUsername,
ONVIFPassword: onvifCredentials.ONVIFPassword,
ONVIFChannel: onvifCredentials.ONVIFChannel,
},
},
},
@@ -644,7 +587,6 @@ func DoTriggerRelayOutput(c *gin.Context) {
ONVIFXAddr: onvifCredentials.ONVIFXAddr,
ONVIFUsername: onvifCredentials.ONVIFUsername,
ONVIFPassword: onvifCredentials.ONVIFPassword,
ONVIFChannel: onvifCredentials.ONVIFChannel,
},
},
},
@@ -1,81 +0,0 @@
package http
import (
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/gin-gonic/gin"
"github.com/kerberos-io/agent/machinery/src/encryption"
"github.com/kerberos-io/agent/machinery/src/models"
)
func TestFilesServesFlatAndDatedRecordings(t *testing.T) {
gin.SetMode(gin.TestMode)
for _, layout := range []string{"flat", "dated"} {
for _, encrypted := range []bool{false, true} {
name := layout + "-plain"
if encrypted {
name = layout + "-encrypted"
}
t.Run(name, func(t *testing.T) {
const fileName = "1767355200123-device-key.mp4"
directory := t.TempDir()
relativePath := fileName
if layout == "dated" {
relativePath = "2026/01/02/" + fileName
}
filePath := filepath.Join(directory, "data/recordings", relativePath)
if err := os.MkdirAll(filepath.Dir(filePath), 0o755); err != nil {
t.Fatal(err)
}
config := &models.Configuration{Config: models.Config{Encryption: &models.Encryption{}}}
content := []byte("recording-payload")
if encrypted {
config.Config.Encryption.Recordings = "true"
config.Config.Encryption.SymmetricKey = "0123456789abcdef0123456789abcdef"
ciphertext, err := encryption.AesEncrypt(content, config.Config.Encryption.SymmetricKey)
if err != nil {
t.Fatal(err)
}
content = []byte(ciphertext)
}
if err := os.WriteFile(filePath, content, 0o644); err != nil {
t.Fatal(err)
}
router := gin.New()
router.GET("/file/*filepath", func(c *gin.Context) { Files(c, directory, config) })
for _, reference := range []string{relativePath, fileName} {
response := httptest.NewRecorder()
router.ServeHTTP(response, httptest.NewRequest("GET", "/file/"+reference, nil))
if response.Code != 200 || response.Body.String() != "recording-payload" {
t.Fatalf("GET %s = %d %q", reference, response.Code, response.Body.String())
}
if got := response.Header().Get("Content-Type"); got != "video/mp4" {
t.Fatalf("Content-Type = %q, want video/mp4", got)
}
}
})
}
}
}
func TestFilesRejectsMissingAndOutsideRecordings(t *testing.T) {
directory := t.TempDir()
if err := os.MkdirAll(filepath.Join(directory, "data/recordings"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(directory, "data/outside.mp4"), []byte("not a recording"), 0o644); err != nil {
t.Fatal(err)
}
router := gin.New()
router.GET("/file/*filepath", func(c *gin.Context) { Files(c, directory, &models.Configuration{}) })
for _, reference := range []string{"missing.mp4", "../outside.mp4", "2026/01/02/missing.mp4"} {
response := httptest.NewRecorder()
router.ServeHTTP(response, httptest.NewRequest("GET", "/file/"+reference, nil))
if response.Code != 404 {
t.Fatalf("GET %s = %d, want 404", reference, response.Code)
}
}
}
-1
View File
@@ -108,7 +108,6 @@ func AddRoutes(r *gin.Engine, authMiddleware *jwt.GinJWTMiddleware, configDirect
api.POST("/camera/onvif/gotopreset", GoToOnvifPreset)
api.POST("/camera/onvif/pantilt", DoOnvifPanTilt)
api.POST("/camera/onvif/zoom", DoOnvifZoom)
api.POST("/camera/onvif/focus", DoOnvifFocus)
api.POST("/camera/onvif/inputs", DoGetDigitalInputs)
api.POST("/camera/onvif/outputs", DoGetRelayOutputs)
api.POST("/camera/onvif/outputs/:output", DoTriggerRelayOutput)
+2 -4
View File
@@ -5,7 +5,6 @@ import (
"io"
"os"
"strconv"
"strings"
"time"
jwt "github.com/appleboy/gin-jwt/v2"
@@ -20,7 +19,6 @@ import (
"github.com/kerberos-io/agent/machinery/src/capture"
"github.com/kerberos-io/agent/machinery/src/encryption"
"github.com/kerberos-io/agent/machinery/src/models"
"github.com/kerberos-io/agent/machinery/src/utils"
swaggerFiles "github.com/swaggo/files"
ginSwagger "github.com/swaggo/gin-swagger"
)
@@ -162,9 +160,9 @@ func requestLogger() gin.HandlerFunc {
func Files(c *gin.Context, configDirectory string, configuration *models.Configuration) {
// Get File
filePath, err := utils.ResolveRecordingPath(configDirectory+"/data/recordings", strings.TrimPrefix(c.Param("filepath"), "/"))
filePath := configDirectory + "/data/recordings" + c.Param("filepath")
_, err := os.Open(filePath)
if err != nil {
log.WithError(err).Warn("Could not resolve recording for playback")
c.JSON(404, gin.H{"error": "File not found"})
return
}
+80 -45
View File
@@ -191,32 +191,51 @@ func GetMediaFormatted(files []os.FileInfo, recordingDirectory string, configura
count := 0
for _, file := range files {
fileName := file.Name()
recordedAt, err := RecordingStartTime(fileName)
if err != nil {
continue
}
timestamp := recordedAt.Unix()
if eventFilter.TimestampOffsetStart > 0 && timestamp < eventFilter.TimestampOffsetStart {
continue
}
if eventFilter.TimestampOffsetEnd > 0 && timestamp >= eventFilter.TimestampOffsetEnd {
continue
}
loc, _ := time.LoadLocation(configuration.Config.Timezone)
localTime := recordedAt.In(loc)
filePaths = append(filePaths, models.Media{
Key: fileName,
Path: recordingDirectory + "/" + fileName,
CameraName: configuration.Config.Name,
CameraKey: configuration.Config.Key,
Day: localTime.Format("02-01-2006"),
ShortDay: localTime.Format("Jan _2"),
Time: localTime.Format("15:04:05"),
Timestamp: strconv.FormatInt(timestamp, 10),
})
count++
if eventFilter.NumberOfElements > 0 && count >= eventFilter.NumberOfElements {
break
fileParts := strings.Split(fileName, "_")
if len(fileParts) == 6 {
timestamp := fileParts[0]
timestampInt, err := strconv.ParseInt(timestamp, 10, 64)
if err == nil {
if eventFilter.TimestampOffsetStart > 0 {
// TimestampOffsetStart represents the newest lower bound to include.
if timestampInt < eventFilter.TimestampOffsetStart {
continue
}
}
// If we have an offset we will check if we should skip or not
if eventFilter.TimestampOffsetEnd > 0 {
// Medias are sorted from new to older. TimestampOffsetEnd holds the oldest
// timestamp of the previous batch of events. By doing this check, we make sure
// to skip the previous batch.
if timestampInt >= eventFilter.TimestampOffsetEnd {
continue
}
}
loc, _ := time.LoadLocation(configuration.Config.Timezone)
time := time.Unix(timestampInt, 0).In(loc)
day := time.Format("02-01-2006")
timeString := time.Format("15:04:05")
shortDay := time.Format("Jan _2")
media := models.Media{
Key: fileName,
Path: recordingDirectory + "/" + fileName,
CameraName: configuration.Config.Name,
CameraKey: configuration.Config.Key,
Day: day,
ShortDay: shortDay,
Time: timeString,
Timestamp: timestamp,
}
filePaths = append(filePaths, media)
count = count + 1
if eventFilter.NumberOfElements > 0 && count >= eventFilter.NumberOfElements {
break
}
}
}
}
return filePaths
@@ -225,12 +244,18 @@ func GetMediaFormatted(files []os.FileInfo, recordingDirectory string, configura
func GetDays(files []os.FileInfo, recordingDirectory string, configuration *models.Configuration) []string {
days := []string{}
for _, file := range files {
recordedAt, err := RecordingStartTime(file.Name())
if err != nil {
continue
fileName := file.Name()
fileParts := strings.Split(fileName, "_")
if len(fileParts) == 6 {
timestamp := fileParts[0]
timestampInt, err := strconv.ParseInt(timestamp, 10, 64)
if err == nil {
loc, _ := time.LoadLocation(configuration.Config.Timezone)
time := time.Unix(timestampInt, 0).In(loc)
day := time.Format("02-01-2006")
days = append(days, day)
}
}
loc, _ := time.LoadLocation(configuration.Config.Timezone)
days = append(days, recordedAt.In(loc).Format("02-01-2006"))
}
uniqueDays := Unique(days)
return uniqueDays
@@ -253,13 +278,10 @@ func NumberOfFilesInDirectory(path string) int {
return len(files)
}
// NumberOfMP4sInDirectory counts MP4 files, including dated subdirectories.
// NumberOfMP4sInDirectory returns the count of all files with mp4 extension in current directory
func NumberOfMP4sInDirectory(path string) int {
files, err := ReadRecordingDirectory(path)
if err != nil {
log.WithError(err).Error("Could not count recordings")
return 0
}
pattern := filepath.Join(path, "*.mp4")
files, _ := filepath.Glob(pattern)
return len(files)
}
@@ -387,13 +409,25 @@ func Decrypt(directoryOrFile string, symmetricKey []byte) {
var files []string
if fileInfo.IsDir() {
dir, err := ReadRecordingDirectory(directoryOrFile)
// Create decrypted directory
err = os.MkdirAll(directoryOrFile+"/decrypted", 0755)
if err != nil {
log.Fatal(err.Error())
return
}
dir, err := os.ReadDir(directoryOrFile)
if err != nil {
log.Fatal(err.Error())
return
}
for _, file := range dir {
files = append(files, filepath.Join(directoryOrFile, file.Name()))
// Check if file is not a directory
if !file.IsDir() {
// Check if an mp4 file
if strings.HasSuffix(file.Name(), ".mp4") {
files = append(files, directoryOrFile+"/"+file.Name())
}
}
}
} else {
files = append(files, directoryOrFile)
@@ -415,12 +449,13 @@ func Decrypt(directoryOrFile string, symmetricKey []byte) {
return
}
outputDirectory := filepath.Join(filepath.Dir(file), "decrypted")
if err := os.MkdirAll(outputDirectory, 0o755); err != nil {
log.Fatal(err.Error())
return
}
err = os.WriteFile(filepath.Join(outputDirectory, filepath.Base(file)), []byte(decrypted), 0644)
// Write decrypted content to file with appended .decrypted
// Get filename split by / and get last element.
fileParts := strings.Split(file, "/")
fileName := fileParts[len(fileParts)-1]
pathToFile := strings.Join(fileParts[:len(fileParts)-1], "/")
err = os.WriteFile(pathToFile+"/decrypted/"+fileName, []byte(decrypted), 0644)
if err != nil {
log.Fatal(err.Error())
return
-146
View File
@@ -1,146 +0,0 @@
package utils
import (
"fmt"
"io/fs"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
func RecordingFileName(timestamp int64, deviceKey string) (string, error) {
if timestamp <= 0 {
return "", fmt.Errorf("invalid recording timestamp %d", timestamp)
}
if deviceKey == "" || strings.ContainsAny(deviceKey, "/\\") {
return "", fmt.Errorf("missing or invalid recording device key")
}
return strconv.FormatInt(timestamp, 10) + "-" + deviceKey + ".mp4", nil
}
func RecordingStartTime(name string) (time.Time, error) {
base := filepath.Base(name)
if !strings.HasSuffix(base, ".mp4") {
return time.Time{}, fmt.Errorf("invalid recording filename %q", name)
}
timestamp, deviceKey, short := strings.Cut(strings.TrimSuffix(base, ".mp4"), "-")
if short && deviceKey != "" {
if millis, err := strconv.ParseInt(timestamp, 10, 64); err == nil && millis > 0 {
return time.UnixMilli(millis), nil
}
}
parts := strings.Split(base, "_")
if len(parts) == 6 {
seconds, err := strconv.ParseInt(parts[0], 10, 64)
if err == nil && seconds > 0 {
return time.Unix(seconds, 0), nil
}
}
return time.Time{}, fmt.Errorf("invalid recording filename %q", name)
}
func CreateRecordingPath(directory, fileName string, recordedAt time.Time) (string, error) {
if !filepath.IsLocal(fileName) || fileName == "." || filepath.Base(fileName) != fileName || strings.ContainsAny(fileName, "/\\") {
return "", fmt.Errorf("invalid recording filename %q", fileName)
}
if recordedAt.UnixMilli() <= 0 {
return "", fmt.Errorf("invalid recording start time %s", recordedAt)
}
directory = filepath.Join(directory, recordedAt.UTC().Format("2006/01/02"))
if err := os.MkdirAll(directory, 0o755); err != nil {
return "", fmt.Errorf("create recording directory: %w", err)
}
return filepath.Join(directory, fileName), nil
}
// ResolveRecordingPath accepts relative paths from listings and historical
// basename-only references from upload markers and playback URLs.
func ResolveRecordingPath(directory, name string) (string, error) {
if !filepath.IsLocal(name) {
return "", fmt.Errorf("invalid recording path %q", name)
}
filePath := filepath.Join(directory, name)
info, err := os.Stat(filePath)
if os.IsNotExist(err) && filepath.Base(name) == name {
recordedAt, parseErr := RecordingStartTime(name)
if parseErr == nil {
filePath = filepath.Join(directory, recordedAt.UTC().Format("2006/01/02"), name)
info, err = os.Stat(filePath)
}
}
if err != nil {
return "", err
}
if !info.Mode().IsRegular() {
return "", fmt.Errorf("recording %q is not a regular file", name)
}
return filePath, nil
}
func OpenRecording(directory, name string) (*os.File, error) {
filePath, err := ResolveRecordingPath(directory, name)
if err != nil {
return nil, err
}
return os.Open(filePath)
}
func RemoveRecording(directory, name string) error {
filePath, err := ResolveRecordingPath(directory, name)
if err != nil {
return err
}
return os.Remove(filePath)
}
type recordingFileInfo struct {
os.FileInfo
relativePath string
}
func (file recordingFileInfo) Name() string {
return file.relativePath
}
// ReadRecordingDirectory returns MP4s with root-relative names, preserving the
// file information used by the existing sorting and media formatting helpers.
func ReadRecordingDirectory(directory string) ([]os.FileInfo, error) {
return readRecordingDirectory(directory, filepath.WalkDir)
}
func readRecordingDirectory(directory string, walkDir func(string, fs.WalkDirFunc) error) ([]os.FileInfo, error) {
files := []os.FileInfo{}
err := walkDir(directory, func(filePath string, entry fs.DirEntry, err error) error {
if err != nil {
if os.IsNotExist(err) && filePath != directory {
return nil
}
return err
}
if entry.IsDir() && entry.Name() == "decrypted" && filePath != directory {
return filepath.SkipDir
}
if !entry.Type().IsRegular() || filepath.Ext(entry.Name()) != ".mp4" {
return nil
}
info, err := entry.Info()
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
name, err := filepath.Rel(directory, filePath)
if err != nil {
return err
}
files = append(files, recordingFileInfo{FileInfo: info, relativePath: filepath.ToSlash(name)})
return nil
})
if err != nil {
return nil, err
}
return files, nil
}
-311
View File
@@ -1,311 +0,0 @@
package utils
import (
"errors"
"io"
"io/fs"
"os"
"path/filepath"
"reflect"
"strconv"
"testing"
"time"
"github.com/kerberos-io/agent/machinery/src/encryption"
"github.com/kerberos-io/agent/machinery/src/models"
)
func TestShortRecordingNamePreservesMillisecondsAndDeviceKey(t *testing.T) {
const timestamp = int64(1767355200123)
const deviceKey = "device-key_with-hyphens"
name, err := RecordingFileName(timestamp, deviceKey)
if err != nil || name != "1767355200123-device-key_with-hyphens.mp4" {
t.Fatalf("RecordingFileName() = %q, %v", name, err)
}
other, err := RecordingFileName(timestamp+1, deviceKey)
if err != nil || other == name {
t.Fatalf("recordings within the same second must have distinct names: %q, %v", other, err)
}
directory := t.TempDir()
filePath, err := CreateRecordingPath(directory, name, time.UnixMilli(timestamp))
if err != nil {
t.Fatal(err)
}
if want := filepath.Join(directory, "2026/01/02", name); filePath != want {
t.Fatalf("path = %q, want %q", filePath, want)
}
if err := os.WriteFile(filePath, []byte("video"), 0o644); err != nil {
t.Fatal(err)
}
resolved, err := ResolveRecordingPath(directory, name)
if err != nil || resolved != filePath {
t.Fatalf("ResolveRecordingPath() = %q, %v", resolved, err)
}
files, err := ReadRecordingDirectory(directory)
if err != nil {
t.Fatal(err)
}
config := &models.Configuration{Config: models.Config{Timezone: "UTC"}}
media := GetMediaFormatted(files, directory, config, models.EventFilter{TimestampOffsetStart: timestamp / 1000})
if len(media) != 1 || media[0].Key != "2026/01/02/"+name || media[0].Timestamp != "1767355200" {
t.Fatalf("unexpected short-name media: %+v", media)
}
if days := GetDays(files, directory, config); !reflect.DeepEqual(days, []string{"02-01-2026"}) {
t.Fatalf("unexpected short-name days: %v", days)
}
if err := RemoveRecording(directory, name); err != nil {
t.Fatal(err)
}
for _, invalidKey := range []string{"", "camera/key", `camera\key`} {
if _, err := RecordingFileName(timestamp, invalidKey); err == nil {
t.Errorf("accepted invalid device key %q", invalidKey)
}
}
if _, err := RecordingFileName(0, deviceKey); err == nil {
t.Fatal("accepted invalid recording timestamp")
}
}
func TestCreateRecordingPathUsesRecordingUTCDate(t *testing.T) {
directory := t.TempDir()
recordedAt := time.Date(2026, time.January, 1, 1, 0, 0, 0, time.FixedZone("UTC+2", 2*60*60))
name := strconv.FormatInt(recordedAt.Unix(), 10) + "_3-123_camera_0-0-0-0_1_1000.mp4"
filePath, err := CreateRecordingPath(directory, name, recordedAt)
if err != nil {
t.Fatal(err)
}
if want := filepath.Join(directory, "2025/12/31", name); filePath != want {
t.Fatalf("path = %q, want %q", filePath, want)
}
if err := os.WriteFile(filePath, []byte("video"), 0o644); err != nil {
t.Fatal(err)
}
for _, reference := range []string{name, "2025/12/31/" + name} {
got, err := ResolveRecordingPath(directory, reference)
if err != nil || got != filePath {
t.Fatalf("ResolveRecordingPath(%q) = %q, %v; want %q", reference, got, err, filePath)
}
file, err := OpenRecording(directory, reference)
if err != nil {
t.Fatal(err)
}
content, err := io.ReadAll(file)
file.Close()
if err != nil || string(content) != "video" {
t.Fatalf("recording content = %q, %v", content, err)
}
}
if err := RemoveRecording(directory, name); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filePath); !os.IsNotExist(err) {
t.Fatalf("recording still exists after removal: %v", err)
}
}
func TestRecordingPathsKeepFlatFilesAccessible(t *testing.T) {
directory := t.TempDir()
name := "legacy-recording.mp4"
filePath := filepath.Join(directory, name)
if err := os.WriteFile(filePath, []byte("legacy"), 0o644); err != nil {
t.Fatal(err)
}
got, err := ResolveRecordingPath(directory, name)
if err != nil || got != filePath {
t.Fatalf("ResolveRecordingPath() = %q, %v; want %q", got, err, filePath)
}
if err := RemoveRecording(directory, name); err != nil {
t.Fatal(err)
}
}
func TestRecordingPathsReportInvalidAndMissingPaths(t *testing.T) {
directory := t.TempDir()
for _, name := range []string{"", ".", "..", "../outside.mp4", "/outside.mp4", "nested/recording.mp4", `nested\recording.mp4`} {
if _, err := CreateRecordingPath(directory, name, time.Now()); err == nil {
t.Errorf("CreateRecordingPath(%q) should reject invalid filename", name)
}
}
for _, name := range []string{"", ".", "../outside.mp4", "/outside.mp4"} {
if _, err := ResolveRecordingPath(directory, name); err == nil {
t.Errorf("ResolveRecordingPath(%q) should reject invalid path", name)
}
}
for _, name := range []string{"missing.mp4", "1700000000_missing.mp4", "2026/01/01/missing.mp4"} {
if _, err := ResolveRecordingPath(directory, name); !os.IsNotExist(err) {
t.Errorf("ResolveRecordingPath(%q) error = %v, want not-exist", name, err)
}
}
blocked := filepath.Join(directory, "not-a-directory")
if err := os.WriteFile(blocked, nil, 0o644); err != nil {
t.Fatal(err)
}
if _, err := CreateRecordingPath(blocked, "recording.mp4", time.Now()); err == nil {
t.Fatal("expected directory creation error")
}
if _, err := CreateRecordingPath(directory, "recording.mp4", time.Time{}); err == nil {
t.Fatal("expected invalid recording start time error")
}
if _, err := ReadRecordingDirectory(filepath.Join(directory, "missing")); !os.IsNotExist(err) {
t.Fatalf("ReadRecordingDirectory() error = %v, want not-exist", err)
}
}
func TestReadRecordingDirectoryIncludesFlatAndDatedMedia(t *testing.T) {
directory := t.TempDir()
names := []string{
"1700000000_6_7_8_9_10.mp4",
"2026/01/02/1767355200_6_7_8_9_10.mp4",
}
for index, name := range names {
filePath := filepath.Join(directory, name)
if err := os.MkdirAll(filepath.Dir(filePath), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filePath, []byte("video"), 0o644); err != nil {
t.Fatal(err)
}
modified := time.Unix(int64(index+1), 0)
if err := os.Chtimes(filePath, modified, modified); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(filepath.Join(directory, "2026/01/02/ignore.txt"), nil, 0o644); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(directory, "2026/01/02/decrypted"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(directory, "2026/01/02/decrypted/output.mp4"), nil, 0o644); err != nil {
t.Fatal(err)
}
files, err := ReadRecordingDirectory(directory)
if err != nil {
t.Fatal(err)
}
if len(files) != 2 || NumberOfMP4sInDirectory(directory) != 2 {
t.Fatalf("expected 2 MP4 recordings, got %d", len(files))
}
config := &models.Configuration{Config: models.Config{Timezone: "UTC", Name: "camera", Key: "device"}}
files = GetSortedDirectory(files)
media := GetMediaFormatted(files, directory, config, models.EventFilter{})
if len(media) != 2 || media[0].Key != names[1] || media[1].Key != names[0] {
t.Fatalf("unexpected media ordering or relative keys: %+v", media)
}
if media[0].Path != filepath.Join(directory, names[1]) || media[0].Timestamp != "1767355200" {
t.Fatalf("unexpected nested media path or timestamp: %+v", media[0])
}
if got, want := GetDays(files, directory, config), []string{"02-01-2026", "14-11-2023"}; !reflect.DeepEqual(got, want) {
t.Fatalf("days = %v, want %v", got, want)
}
filtered := GetMediaFormatted(files, directory, config, models.EventFilter{TimestampOffsetStart: 1767355200})
if len(filtered) != 1 || filtered[0].Key != names[1] {
t.Fatalf("unexpected filtered nested media: %+v", filtered)
}
}
func TestReadRecordingDirectoryContinuesAfterConcurrentDeletion(t *testing.T) {
for _, origin := range []string{"Info", "WalkDir"} {
t.Run(origin, func(t *testing.T) {
directory := t.TempDir()
for _, name := range []string{"deleted.mp4", "surviving.mp4"} {
if err := os.WriteFile(filepath.Join(directory, name), []byte("video"), 0o644); err != nil {
t.Fatal(err)
}
}
entries, err := os.ReadDir(directory)
if err != nil {
t.Fatal(err)
}
files, err := readRecordingDirectory(directory, func(root string, visit fs.WalkDirFunc) error {
deleted := filepath.Join(root, entries[0].Name())
if err := os.Remove(deleted); err != nil {
t.Fatal(err)
}
var err error
if origin == "Info" {
err = visit(deleted, entries[0], nil)
} else {
err = visit(deleted, nil, &fs.PathError{Op: "lstat", Path: deleted, Err: fs.ErrNotExist})
}
if err != nil {
return err
}
return visit(filepath.Join(root, entries[1].Name()), entries[1], nil)
})
if err != nil {
t.Fatal(err)
}
if len(files) != 1 || files[0].Name() != "surviving.mp4" || files[0].Size() != 5 {
t.Fatalf("expected surviving recording after concurrent deletion, got %v", files)
}
})
}
}
type recordingDirEntryInfoError struct {
fs.DirEntry
err error
}
func (entry recordingDirEntryInfoError) Info() (fs.FileInfo, error) {
return nil, entry.err
}
func TestReadRecordingDirectoryReportsOtherScanErrors(t *testing.T) {
for _, origin := range []string{"Info", "WalkDir"} {
t.Run(origin, func(t *testing.T) {
directory := t.TempDir()
filePath := filepath.Join(directory, "recording.mp4")
if err := os.WriteFile(filePath, []byte("video"), 0o644); err != nil {
t.Fatal(err)
}
entries, err := os.ReadDir(directory)
if err != nil {
t.Fatal(err)
}
wantErr := &fs.PathError{Op: "lstat", Path: filePath, Err: fs.ErrPermission}
files, err := readRecordingDirectory(directory, func(root string, visit fs.WalkDirFunc) error {
if origin == "Info" {
return visit(filePath, recordingDirEntryInfoError{DirEntry: entries[0], err: wantErr}, nil)
}
return visit(filePath, nil, wantErr)
})
if !errors.Is(err, fs.ErrPermission) || files != nil {
t.Fatalf("scan = %v, %v; want permission error without partial results", files, err)
}
})
}
}
func TestDecryptDirectoryIncludesDatedRecordingsAndSkipsOutputs(t *testing.T) {
directory := t.TempDir()
key := []byte("0123456789abcdef0123456789abcdef")
payload := []byte("recording-payload")
ciphertext, err := encryption.AesEncrypt(payload, string(key))
if err != nil {
t.Fatal(err)
}
names := []string{"legacy.mp4", "2026/01/02/recording.mp4"}
for _, name := range names {
filePath := filepath.Join(directory, name)
if err := os.MkdirAll(filepath.Dir(filePath), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filePath, []byte(ciphertext), 0o644); err != nil {
t.Fatal(err)
}
}
Decrypt(directory, key)
Decrypt(directory, key)
for _, name := range names {
output := filepath.Join(directory, filepath.Dir(name), "decrypted", filepath.Base(name))
content, err := os.ReadFile(output)
if err != nil || string(content) != string(payload) {
t.Fatalf("decrypted %q = %q, %v", name, content, err)
}
}
if got := NumberOfMP4sInDirectory(directory); got != len(names) {
t.Fatalf("recording count = %d, want %d (excluding decrypted outputs)", got, len(names))
}
}