refactor(hub): one sampling model, follow-only downstream

Downstream services (hub-workflows, pipe-*, workflow stages) only continue
traces started at the Vault or hub-api, so they now always render
OTEL_TRACES_SAMPLER=parentbased_always_off when OpenTelemetry is enabled
instead of a free-form opentelemetry.sampler, which a non-parent-based value
would have turned into independent re-sampling. hub-api is configured only
through opentelemetry.apiSamplingRatios (TRACE_SAMPLER_RATIO_<CLASS>);
opentelemetry.apiSampler is removed. Neither removed value is used by any
environment except production's sampler.type: parentbased_always_off, which
is now the fixed behaviour, so production renders unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

charts/hub/Chart.yaml

@@ -16,7 +16,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.164.0
+version: 0.165.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

charts/hub/README.md

@@ -268,10 +268,8 @@ kerberospipeline:
 | `opentelemetry.enabled` | Enable or disable OpenTelemetry instrumentation. | `false` |
 | `opentelemetry.routingEnabled` | Enable or disable OpenTelemetry routing/export behavior. | `false` |
 | `opentelemetry.collector.endpoint` | OpenTelemetry collector endpoint used for trace export. | `"http://otel-collector:4317"` |
-| `opentelemetry.sampler.type` | Trace sampler for the services downstream of the Vault (hub-workflows, `pipe-*`, workflow stages), exported as `OTEL_TRACES_SAMPLER`. They continue the Vault's trace, so use `parentbased_always_off` and sample at the Vault. Empty keeps the SDK default (sample everything). | `""` |
-| `opentelemetry.sampler.arg` | Sampler argument for those services, exported as `OTEL_TRACES_SAMPLER_ARG`. | `""` |
-| `opentelemetry.apiSampler.type` | Trace sampler for hub-api, which starts its own traces for user requests (e.g. `parentbased_traceidratio`). Empty keeps the SDK default. | `""` |
-| `opentelemetry.apiSampler.arg` | Sampler argument for hub-api (e.g. `"0.1"` for ~10% of requests). | `""` |
+| `opentelemetry.apiSamplingRatios.operator` | hub-api: share of user and API-client requests traced (0–1), rendered as `TRACE_SAMPLER_RATIO_OPERATOR` (hub-api on trace v1.2.5+). With no ratio set, hub-api records every trace. hub-workflows, the `pipe-*` services and the workflow stages always follow the trace they continue (`parentbased_always_off`); uploads are sampled at the Vault. | `""` |
+| `opentelemetry.apiSamplingRatios.default` | hub-api: share of everything else traced (device heartbeats, agent uploads and live segments, background work), rendered as `TRACE_SAMPLER_RATIO_DEFAULT`; `0` turns it off. | `""` |
 | `openai.enabled` | Enable or disable OpenAI-backed semantic features. | `false` |
 | `openai.apikey` | OpenAI API key used when OpenAI integration is enabled. | `"xxx"` |
 | `kerberosvault.uri` | The default Kerberos Vault uri (you can add multiple within the app), by default `'https://api.storage.yourdomain.com'` | `"https://api.vault.yourdomain.com"` |

charts/hub/templates/_helpers.tpl

@@ -84,30 +84,25 @@ Usage: {{- include "hub.mongodb.appUriEnv" (dict "root" $ "app" "hub-api") | nin
 {{- end -}}
 
 {{/*
-OpenTelemetry sampler env entries, rendered only when configured. The OTel SDKs
-read these natively; unset keeps the SDK default (parentbased_always_on).
-"hub.opentelemetry.samplerEnv" renders opentelemetry.sampler, the sampler of the
-services downstream of the Vault (pipeline, workflows engine and stages), which
-continue the Vault's trace; "hub.opentelemetry.apiSamplerEnv" renders
-opentelemetry.apiSampler for hub-api, which starts its own traces for user
-requests.
+OpenTelemetry sampling env.
+"hub.opentelemetry.samplerEnv" is for the services that only continue traces
+started elsewhere (hub-workflows, pipe-*, workflow stages): they follow the
+parent's sampling decision and never start traces of their own.
+"hub.opentelemetry.apiSamplerEnv" renders hub-api's per-class sampling ratios
+(opentelemetry.apiSamplingRatios) as TRACE_SAMPLER_RATIO_<CLASS>; empty values
+are not rendered.
 Usage: {{- include "hub.opentelemetry.samplerEnv" $ | trim | nindent 8 }}
 */}}
 {{- define "hub.opentelemetry.samplerEnv" -}}
-{{- include "hub.opentelemetry.samplerEnvFrom" .Values.opentelemetry.sampler -}}
+- name: OTEL_TRACES_SAMPLER
+  value: "parentbased_always_off"
 {{- end -}}
 
 {{- define "hub.opentelemetry.apiSamplerEnv" -}}
-{{- include "hub.opentelemetry.samplerEnvFrom" (.Values.opentelemetry.apiSampler | default dict) -}}
-{{- end -}}
-
-{{- define "hub.opentelemetry.samplerEnvFrom" -}}
-{{- with .type }}
-- name: OTEL_TRACES_SAMPLER
-  value: {{ . | quote }}
+{{- range $class, $ratio := (.Values.opentelemetry.apiSamplingRatios | default dict) }}
+{{- if ne (toString $ratio) "" }}
+- name: TRACE_SAMPLER_RATIO_{{ upper $class }}
+  value: {{ toString $ratio | quote }}
 {{- end }}
-{{- with .arg }}
-- name: OTEL_TRACES_SAMPLER_ARG
-  value: {{ . | quote }}
 {{- end }}
 {{- end -}}

charts/hub/values.yaml

@@ -209,28 +209,27 @@ opentelemetry:
     # the collector terminates TLS. A scheme-less value defaults to TLS and
     # fails against a plaintext collector ("server gave HTTP response to HTTPS client").
     endpoint: "http://otel-collector:4318"
-  # Trace sampling, passed to the services as OTEL_TRACES_SAMPLER and
-  # OTEL_TRACES_SAMPLER_ARG. Leave empty to keep the SDK default, which records
-  # every trace.
-  # Supported types: always_on, always_off, traceidratio, parentbased_always_on,
-  # parentbased_always_off, parentbased_traceidratio.
+  # Trace sampling is decided where a trace starts:
+  # - Uploads start at the Vault, whose deployment sets their rate
+  #   (TRACE_SAMPLER_RATIO_UPLOAD / TRACE_SAMPLER_RATIO_DEFAULT).
+  # - hub-api starts its own traces; apiSamplingRatios below sets its rates.
+  # hub-workflows, the pipe-* services and the workflow stages only continue
+  # traces started elsewhere: they follow that decision and never start traces
+  # of their own (OTEL_TRACES_SAMPLER=parentbased_always_off, not configurable).
   #
-  # sampler applies to the services downstream of the Vault: hub-workflows, the
-  # pipe-* services and the workflow stages. They continue the trace the Vault
-  # starts for an upload, so head sampling belongs at the Vault
-  # (parentbased_traceidratio on the Vault deployment) and these follow its
-  # decision without starting traces of their own:
-  #   type: "parentbased_always_off"
-  sampler:
-    type: ""
-    arg: ""
-  # apiSampler applies to hub-api, which starts its own traces for user
-  # requests. Example: keep ~10% of them:
-  #   type: "parentbased_traceidratio"
-  #   arg: "0.1"
-  apiSampler:
-    type: ""
-    arg: ""
+  # apiSamplingRatios: hub-api's sampling ratio (0 to 1) per kind of work,
+  # rendered as TRACE_SAMPLER_RATIO_<CLASS> (hub-api on trace library v1.2.5+).
+  # Empty values are not rendered; with none set, hub-api records every trace.
+  #   operator: requests from users and API clients (UI, API tokens).
+  #   default:  everything else hub-api starts: device heartbeats, agent uploads
+  #             and live segments, background work, queries outside a request.
+  #             Unset with operator set: records everything.
+  # Example: ~10% of user requests, 1% of everything else:
+  #   operator: "0.1"
+  #   default: "0.01"
+  apiSamplingRatios:
+    operator: ""
+    default: ""
 ############################################
 # OpenAI configuration (semantic search)
 #