refactor(hub): one sampling model, follow-only downstream

Downstream services (hub-workflows, pipe-*, workflow stages) only continue
traces started at the Vault or hub-api, so they now always render
OTEL_TRACES_SAMPLER=parentbased_always_off when OpenTelemetry is enabled
instead of a free-form opentelemetry.sampler, which a non-parent-based value
would have turned into independent re-sampling. hub-api is configured only
through opentelemetry.apiSamplingRatios (TRACE_SAMPLER_RATIO_<CLASS>);
opentelemetry.apiSampler is removed. Neither removed value is used by any
environment except production's sampler.type: parentbased_always_off, which
is now the fixed behaviour, so production renders unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

charts/hub/Chart.yaml

@@ -16,7 +16,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.162.0
+version: 0.165.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

charts/hub/README.md

@@ -268,8 +268,8 @@ kerberospipeline:
 | `opentelemetry.enabled` | Enable or disable OpenTelemetry instrumentation. | `false` |
 | `opentelemetry.routingEnabled` | Enable or disable OpenTelemetry routing/export behavior. | `false` |
 | `opentelemetry.collector.endpoint` | OpenTelemetry collector endpoint used for trace export. | `"http://otel-collector:4317"` |
-| `opentelemetry.sampler.type` | Trace sampler, exported as `OTEL_TRACES_SAMPLER` (e.g. `parentbased_traceidratio`). Empty keeps the SDK default (sample everything). | `""` |
-| `opentelemetry.sampler.arg` | Sampler argument, exported as `OTEL_TRACES_SAMPLER_ARG` (e.g. `"0.1"` for ~10% of traces). | `""` |
+| `opentelemetry.apiSamplingRatios.operator` | hub-api: share of user and API-client requests traced (0–1), rendered as `TRACE_SAMPLER_RATIO_OPERATOR` (hub-api on trace v1.2.5+). With no ratio set, hub-api records every trace. hub-workflows, the `pipe-*` services and the workflow stages always follow the trace they continue (`parentbased_always_off`); uploads are sampled at the Vault. | `""` |
+| `opentelemetry.apiSamplingRatios.default` | hub-api: share of everything else traced (device heartbeats, agent uploads and live segments, background work), rendered as `TRACE_SAMPLER_RATIO_DEFAULT`; `0` turns it off. | `""` |
 | `openai.enabled` | Enable or disable OpenAI-backed semantic features. | `false` |
 | `openai.apikey` | OpenAI API key used when OpenAI integration is enabled. | `"xxx"` |
 | `kerberosvault.uri` | The default Kerberos Vault uri (you can add multiple within the app), by default `'https://api.storage.yourdomain.com'` | `"https://api.vault.yourdomain.com"` |
@@ -592,6 +592,7 @@ kerberospipeline:
 | `kerberospipeline.analysis.tag` | The Docker image tag/version. | `"v1.7.8"` |
 | `kerberospipeline.analysis.replicas` | Number of replicas for `kerberospipeline.analysis`. | `1` |
 | `kerberospipeline.analysis.logLevel` | Log verbosity level for `kerberospipeline.analysis`. | `"info"` |
+| `kerberospipeline.analysis.thumbnailRegeneration.enabled` | Regenerate thumbnails from the first frame of a classified moving object (`THUMBNAIL_REGENERATION_ENABLED`). Adds one thumbnail job per classified recording. Requires an analysis image that reads the flag. | `false` |
 | `kerberospipeline.analysis.dbCache.enabled` | Cache hot-path MongoDB lookups (site vault per device) in memory. Site changes apply after at most one TTL. | `false` |
 | `kerberospipeline.analysis.dbCache.ttl` | Lifetime of a cached lookup, as a Go duration (e.g. `"30s"`, `"5m"`). | `"5m"` |
 | `kerberospipeline.analysis.resources.requests.memory` | Memory request for `kerberospipeline.analysis`. | `"10Mi"` |
@@ -681,6 +682,87 @@ kerberospipeline:
 | `kerberoshub.frontend.ssoDomain` | The domain that's being used to activate SSO from the login page. | `""` |
 | `kerberoshub.frontend.stripe.apikey` | If using the public version, `stripe` can be used for automated billing and subscriptions. | `""` |
 
+### Email templates and variables
+
+Every email Hub sends is a template: an HTML file, plus a plain-text `.txt` version, containing `{{variable}}` placeholders that are filled in just before sending. This section lists which value each placeholder gets.
+
+#### Which template is used
+
+- `email.templates.<name>` sets the template name for each email (for example `email.templates.detection: "detection"`).
+- With `email.provider: smtp`, Hub reads `/mail/templates/<name>.html` and `/mail/templates/<name>.txt` from the container. If the files are not there, it uses its built-in template with that name. To use your own, mount a volume at `/mail` (see the commented `custom-email-templates` volume in `values.yaml`). The [`custom-layout/templates`](custom-layout/templates) folder holds a complete set to start from.
+- With `email.provider: mailgun`, `<name>` is a template stored in your Mailgun account. Mailgun receives `user`, `text`, `link`, `numberOfMedia`, `dataUsage` and the email-specific values below as template variables. The computed values (dates, camera, sites, groups, thumbnail) are only filled in with `smtp`.
+- Alert emails (`detection`) are only sent when `email.provider` is `smtp`.
+- A placeholder that has no value for that email stays in the email as literal text, so only use the placeholders listed for it. The bundled templates contain `{{tab1_href}}`, `{{tab1_title}}`, `{{tab2_href}}` and `{{tab2_title}}`; these are not filled in.
+
+#### Links
+
+- The **base URL** is `kerberoshub.frontend.schema` + `://` + `kerberoshub.frontend.url` (the `PUBLIC_URL` environment variable), for example `https://app.example.com`.
+- A **tenant URL** is the base URL with the first part of the host replaced by the recipient account's domain. With base URL `https://app.example.com` and account domain `acme`, the tenant URL is `https://acme.example.com`. Accounts without a domain use the base URL. Only the last two parts of the host are kept, so a base URL on `app.example.co.uk` becomes `acme.co.uk`.
+
+#### Alerts: `detection`
+
+Sent by the notification pipeline (`kerberospipeline.notify`) when a custom alert or the generic detection alert fires. The subject is `email.templates.alertTitle`; for a custom alert it is prefixed with the alert's name in brackets, for example `[Front door] Hub detected an event.`
+
+| Placeholder | Value |
+|---|---|
+| `{{user}}` | Name of the account that receives the alert: first and last name, or the username when both are empty. This is the account the camera belongs to, not the user who created the alert. |
+| `{{title}}` | The email subject. |
+| `{{text}}` | The alert message: what was detected, on which camera and when. |
+| `{{link}}` | **Custom alerts**: a signed link to the recording file. It stops working when the storage link expires. **Generic detection alert**: the recording's page in Hub, `<tenant URL>/media/<id>`. |
+| `{{medialink}}` | `https://app.kerberos.io/media/<id>`. Always this host; it does not follow `kerberoshub.frontend.url`. |
+| `{{thumbnail}}` | An `<img>` of the recording: embedded in the email for custom alerts, a signed thumbnail URL for the generic detection alert. |
+| `{{date}}`, `{{time}}`, `{{datetime}}` | Start of the recording, in the account's timezone. |
+| `{{eventdate}}`, `{{eventtime}}`, `{{eventdatetime}}` | When the alert was sent, in the account's timezone. |
+| `{{timezone}}` | The account's timezone. |
+| `{{devicename}}`, `{{deviceid}}` | Camera name and camera key. |
+| `{{sites}}`, `{{groups}}` | Comma-separated names of the sites and groups the camera belongs to. |
+| `{{classifications}}` | Comma-separated objects detected in the recording. |
+
+#### Camera status: `device`
+
+Sent by `hub-monitor-device` when a camera goes online, offline or idle. The subject is `email.templates.deviceTitle`.
+
+| Placeholder | Value |
+|---|---|
+| `{{user}}` | Username of the account the camera belongs to. |
+| `{{title}}` | The email subject. |
+| `{{text}}` | The status change, for example "`<camera>` is online, but the camera stream is not connected." |
+| `{{status}}` | `online`, `offline` or `idle`. |
+| `{{devicename}}`, `{{deviceid}}` | Camera name and camera key. |
+| `{{eventdate}}`, `{{eventtime}}`, `{{eventdatetime}}` | When the status changed, in the server's timezone (normally UTC), not the account's. |
+| `{{link}}` | Not set; replaced with an empty value. |
+
+#### Account and sharing emails
+
+Sent by the API (`kerberoshub.api`). The subject of each email is the matching `email.templates.<name>Title` value.
+
+| Email (`email.templates.*`) | Placeholder | Value |
+|---|---|---|
+| Registration (`welcome`) | `{{user}}` | New user's first and last name, or username. |
+| | `{{link}}` | Activation link: `<base URL>/login/activate/<token>`. |
+| | `{{token}}` | The activation token. |
+| Account activated (`activate`) | `{{user}}` | User's first and last name, or username. |
+| | `{{link}}` | `<base URL>/login`. |
+| Password reset (`forgot`) | `{{user}}` | User's first and last name, or username. |
+| | `{{password}}` | The newly generated password. |
+| | `{{ipaddress}}` | IP address the reset was requested from. |
+| Recording shared (`share`) | `{{user}}` | Sender's first and last name, or username. |
+| | `{{url}}` | Signed link to the recording file. |
+| | `{{title}}` | The email subject (`email.templates.shareTitle`). The share's own title is only used when that value is empty. |
+| | `{{body}}` | The notes the sender added to the share. |
+| Case shared (`share_case`) | `{{user}}` | Sender's first and last name, or email address. |
+| | `{{firstname}}`, `{{lastname}}` | Sender's first and last name. |
+| | `{{url}}` | `<base URL>/share/<token>`. |
+| | `{{expiry}}` | How long the share stays valid, for example "2 days" or "1 week". |
+| Case share code (`share_case_otp`) | `{{code}}` | One-time code to open the shared case. |
+| | `{{expiry}}` | How long the code stays valid. |
+| Task assigned (`assign_task`) | `{{assignee}}` | Assigned user's first and last name, or username. |
+| | `{{user}}` | Name of the user who assigned the task. |
+| | `{{task_name}}` | The task title. |
+| | `{{link}}` | `<tenant URL>/tasks/`, using the assigned user's domain. |
+
+The `newip`, `disable` and `highupload` templates are included, but no service currently sends them.
+
 ### Post installation
 
 After the installation you'll need to initialise the Mongodb with some objects. Have a look at the `mongodb/` folder, you'll find three files available:

charts/hub/templates/_helpers.tpl

@@ -84,17 +84,25 @@ Usage: {{- include "hub.mongodb.appUriEnv" (dict "root" $ "app" "hub-api") | nin
 {{- end -}}
 
 {{/*
-OpenTelemetry sampler env entries, rendered only when configured. The OTel SDKs
-read these natively; unset keeps the SDK default (parentbased_always_on).
+OpenTelemetry sampling env.
+"hub.opentelemetry.samplerEnv" is for the services that only continue traces
+started elsewhere (hub-workflows, pipe-*, workflow stages): they follow the
+parent's sampling decision and never start traces of their own.
+"hub.opentelemetry.apiSamplerEnv" renders hub-api's per-class sampling ratios
+(opentelemetry.apiSamplingRatios) as TRACE_SAMPLER_RATIO_<CLASS>; empty values
+are not rendered.
 Usage: {{- include "hub.opentelemetry.samplerEnv" $ | trim | nindent 8 }}
 */}}
 {{- define "hub.opentelemetry.samplerEnv" -}}
-{{- with .Values.opentelemetry.sampler.type }}
 - name: OTEL_TRACES_SAMPLER
-  value: {{ . | quote }}
+  value: "parentbased_always_off"
+{{- end -}}
+
+{{- define "hub.opentelemetry.apiSamplerEnv" -}}
+{{- range $class, $ratio := (.Values.opentelemetry.apiSamplingRatios | default dict) }}
+{{- if ne (toString $ratio) "" }}
+- name: TRACE_SAMPLER_RATIO_{{ upper $class }}
+  value: {{ toString $ratio | quote }}
 {{- end }}
-{{- with .Values.opentelemetry.sampler.arg }}
-- name: OTEL_TRACES_SAMPLER_ARG
-  value: {{ . | quote }}
 {{- end }}
 {{- end -}}

charts/hub/templates/kerberos-hub/hub-api.yaml

@@ -597,6 +597,6 @@ spec:
               value: "{{ .Values.opentelemetry.routingEnabled }}"
             - name: OTEL_EXPORTER_OTLP_ENDPOINT
               value: "{{ .Values.opentelemetry.collector.endpoint }}"
-            {{- include "hub.opentelemetry.samplerEnv" $ | trim | nindent 12 }}
+            {{- include "hub.opentelemetry.apiSamplerEnv" $ | trim | nindent 12 }}
             {{- end }}
 {{- end }}

charts/hub/templates/kerberos-pipeline/hub-stage.yaml

@@ -117,6 +117,14 @@ spec:
         - name: KERBEROS_STORAGE_PROVIDER
           value: {{ . | quote }}
         {{- end }}
+        {{- if $root.Values.opentelemetry.enabled }}
+
+        # OpenTelemetry tracing: the stage continues the engine's dispatch span
+        # (the run's traceparent) and follows its sampling decision.
+        - name: OTEL_EXPORTER_OTLP_ENDPOINT
+          value: "{{ $root.Values.opentelemetry.collector.endpoint }}"
+        {{- include "hub.opentelemetry.samplerEnv" $root | trim | nindent 8 }}
+        {{- end }}
 
         # Per-stage tuning knobs. Any key/value under services.<name>.env is
         # rendered verbatim as container env, so a worker can be tuned from

charts/hub/templates/kerberos-pipeline/pipe-analysis.yaml

@@ -126,6 +126,10 @@ spec:
         - name: DOMINANTCOLOR_ENABLED
           value: "{{ .Values.kerberospipeline.dominantColor.enabled }}"
 
+        # Thumbnail regeneration from classified frames (opt-in)
+        - name: THUMBNAIL_REGENERATION_ENABLED
+          value: {{ .Values.kerberospipeline.analysis.thumbnailRegeneration.enabled | default false | quote }}
+
         # In-memory cache for hot-path MongoDB lookups
         - name: DB_CACHE_ENABLED
           value: {{ .Values.kerberospipeline.analysis.dbCache.enabled | default false | quote }}

charts/hub/values.yaml

@@ -209,17 +209,27 @@ opentelemetry:
     # the collector terminates TLS. A scheme-less value defaults to TLS and
     # fails against a plaintext collector ("server gave HTTP response to HTTPS client").
     endpoint: "http://otel-collector:4318"
-  # Trace sampling, passed to the services as OTEL_TRACES_SAMPLER and
-  # OTEL_TRACES_SAMPLER_ARG. Leave empty to keep the SDK default, which records
-  # every trace. Example: keep ~10% of traces, and follow the parent's decision
-  # for spans that continue an upstream trace:
-  #   type: "parentbased_traceidratio"
-  #   arg: "0.1"
-  # Supported types: always_on, always_off, traceidratio, parentbased_always_on,
-  # parentbased_always_off, parentbased_traceidratio.
-  sampler:
-    type: ""
-    arg: ""
+  # Trace sampling is decided where a trace starts:
+  # - Uploads start at the Vault, whose deployment sets their rate
+  #   (TRACE_SAMPLER_RATIO_UPLOAD / TRACE_SAMPLER_RATIO_DEFAULT).
+  # - hub-api starts its own traces; apiSamplingRatios below sets its rates.
+  # hub-workflows, the pipe-* services and the workflow stages only continue
+  # traces started elsewhere: they follow that decision and never start traces
+  # of their own (OTEL_TRACES_SAMPLER=parentbased_always_off, not configurable).
+  #
+  # apiSamplingRatios: hub-api's sampling ratio (0 to 1) per kind of work,
+  # rendered as TRACE_SAMPLER_RATIO_<CLASS> (hub-api on trace library v1.2.5+).
+  # Empty values are not rendered; with none set, hub-api records every trace.
+  #   operator: requests from users and API clients (UI, API tokens).
+  #   default:  everything else hub-api starts: device heartbeats, agent uploads
+  #             and live segments, background work, queries outside a request.
+  #             Unset with operator set: records everything.
+  # Example: ~10% of user requests, 1% of everything else:
+  #   operator: "0.1"
+  #   default: "0.01"
+  apiSamplingRatios:
+    operator: ""
+    default: ""
 ############################################
 # OpenAI configuration (semantic search)
 #
@@ -1277,6 +1287,12 @@ kerberospipeline:
     #  - name: extra
     #    mountPath: /data
     logLevel: "info" # possible values: trace, debug, info, warn, error
+    # Opt-in: after classification, re-enqueue the recording on the thumbnail
+    # queue to regenerate its thumbnail from the first frame of a tracked moving
+    # object (THUMBNAIL_REGENERATION_ENABLED). Adds one thumbnail job per
+    # classified recording; size pipe-thumbnail accordingly before enabling.
+    thumbnailRegeneration:
+      enabled: false
     # In-memory cache for hot-path MongoDB lookups (site vault per device).
     # Site changes apply after at most one TTL; each replica caches separately.
     dbCache: