8 Commits

Author SHA1 Message Date
Cédric Verstraeten
03075a90b8 Merge pull request #9 from kerberos-io/public-release-1739963582
A new public release - 1739963582
2025-02-19 12:14:24 +01:00
uug4ai
e790f713bd A new public release - 1739963582 2025-02-19 11:13:03 +00:00
Cedric Verstraeten
a5afeb6a8d Update README.kustomize.md 2025-02-19 09:26:12 +01:00
Cedric Verstraeten
9d87c58cbb downgrade resources 2025-02-19 08:17:46 +01:00
Cedric Verstraeten
a3dc846ca8 decrease resources 2025-02-19 08:12:12 +01:00
Cedric Verstraeten
45340390cc add kustomize tutorial + configure.md 2025-02-19 07:44:30 +01:00
Cedric Verstraeten
a8a2fb0ea3 add minio nodeports 2025-02-19 00:34:41 +01:00
Cedric Verstraeten
cd67f73700 remove minio, use microk8s instead, bucket creation on deployment 2025-02-19 00:07:18 +01:00
9 changed files with 340 additions and 148 deletions

0
README.configure.md Normal file
View File

View File

@@ -71,15 +71,16 @@ For more detailed instructions and troubleshooting, please refer to the official
## Dependencies
When installing the Kerberos.io stack, several dependencies are required for storage, such as a database (e.g., MongoDB) and a message broker (e.g., RabbitMQ) for asynchronous behavior. We will install these components before setting up the Kerberos Agents and Kerberos Vault.
When deploying the various solutions, several dependencies are essential for storage, including a database (e.g., MongoDB) and a message broker (e.g., RabbitMQ) for asynchronous operations. These components must be installed prior to setting up the Agents, Factory, Vault, and Hub.
One of the key advantages of MicroK8s is its out-of-the-box addons, which can be enabled with a single command. This eliminates the need for complex Helm charts or operators, simplifying the setup process. We will enable some common services, such as DNS, GPU support, and storage, to streamline the installation.
One of the significant advantages of MicroK8s is its built-in addons, which can be enabled with a single command. This feature eliminates the need for complex Helm charts or operators, thereby simplifying the setup process. In this guide, we will enable several common services, such as DNS, GPU support, and storage, to streamline the installation. However, if you prefer more control, you may opt to manually install these components using their respective Helm charts or operators.
```bash
microk8s enable dns
microk8s enable dashboard
microk8s enable nvidia
microk8s enable hostpath-storage
microk8s enable minio
```
You can verify the status of the enabled addons by running the following command:
@@ -94,6 +95,34 @@ Or view the pod status with:
kubectl get po -w -A
```
### Storage class
By default, the `hostpath-storage` module uses a dedicated directory on your filesystem. In most cases, you may prefer to use a dedicated hard drive for storing your recordings, database, and other data. To achieve this, you can create your own storage class and assign it to the desired directory. Create a file `ssd-hostpath-sc.yaml` with following contents.
```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: ssd-hostpath
provisioner: microk8s.io/hostpath
reclaimPolicy: Delete
parameters:
pvDir: /media/Storage
volumeBindingMode: WaitForFirstConsumer
```
Save the previously mentioned file `CTRL+O` and apply the Kubernetes resource.
```bash
kubectl apply -f ssd-hostpath-sc.yaml
```
You can verify the creation of the `storage class` using the following command. This `storage class` will be used in the subsequent installation steps, where each component, such as MongoDB, will create a `persistent volume` using the previously created `storage class`.
```bash
kubectl get sc -A
```
### Clone repository
Next, we will clone this repository to our local environment. This will allow us to execute the necessary configuration files for installing the Minio operator, MongoDB Helm chart, and other required components.
@@ -107,7 +136,55 @@ cd deployment
In contrast to the detailed installation instructions, as mentioned here, an easier option to install is to use our Kustomize configure. This will allow you to specify and create your own overlays to install all the different components through a single command line.
kubectl kustomize overlays/microk8s/ --enable-helm --load-restrictor LoadRestrictionsNone | kubectl apply -f -
Kustomize uses the concept of `bases` and `overlays`, allowing you to customize the base installation with different settings (an overlay). Before executing the `kustomize` command below, navigate to the `overlays/microk8s/kustomization.yaml` file and modify the `inlineValues` of the Hub to match the IP address of your node.
```yaml
valuesInline:
license: "L/+FAw...sJZRBAA"
mqtt:
host: "<ipaddress>"
port: "31080"
protocol: "ws"
...
host: "turn:<ipaddress>:8443"
...
kerberoshub:
api:
url: "<ipaddress>:32081"
```
Run the modified overlay using the following command:
```bash
kubectl kustomize overlays/microk8s/ --enable-helm --load-restrictor LoadRestrictionsNone | kubectl apply -f -
```
Once the installation is running you should see something like following:
```bash
ubuntuvms@ubuntuvms:~/deployment$ kubectl kustomize overlays/microk8s/ --enable-helm --load-restrictor LoadRestrictionsNone | kubectl apply -f -
namespace/kerberos-agent unchanged
namespace/kerberos-factory unchanged
namespace/kerberos-hub unchanged
namespace/kerberos-vault unchanged
namespace/minio-tenant unchanged
namespace/mongodb unchanged
namespace/rabbitmq unchanged
namespace/vernemq unchanged
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com configured
...
```
Verify the installation using the `kubectl` command, it might take some time until all the Kubernetes pods are spinned up. Once everything is stable you should be able to access Factory, Vault and Hub using the node ip address their designated node ports.
Continue with the [`configuration tutorial`](./README.configure.md) to start with the configuration and integration of the various tools.
## Cleanup
@@ -118,18 +195,6 @@ microk8s reset
sudo snap remove microk8s
```
or if you want to keep the Microk8s installation you can also delete the individual deployments.
```bash
kubectl delete -f data-filtering-deployment.yaml
kubectl delete -f kerberos-agent-deployment.yaml
kubectl delete -f ./kerberos-vault-deployment.yaml -n kerberos-vault
kubectl delete -f ./mongodb-config.yaml -n kerberos-vault
helm del rabbitmq -n rabbitmq
helm del mongodb -n mongodb
git clone --depth 1 --branch v6.0.1 https://github.com/minio/operator.git && kubectl delete -k operator/
```
You can confirm all the workloads were removed from your system.
```bash

View File

@@ -27,11 +27,11 @@ spec:
image: "uugai/factory:v1.0.5"
resources:
requests:
memory: 256Mi
cpu: 500m
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 500m
memory: 128Mi
cpu: 100m
ports:
- containerPort: 80
envFrom:

View File

@@ -143,7 +143,7 @@ kerberoshub:
api:
repository: uugai/hub-api
pullPolicy: IfNotPresent
tag: "v1.0.16"
tag: "v1.0.19"
replicas: 1
jwtSecret: "I1JcwzW3A0t_THIS-IS_NOT_A_SECRET_WJK9jnPkipbnVTpf0efMy" # change to a random value, this is for generating JWT tokens.
schema: "http"
@@ -226,7 +226,7 @@ kerberoshub:
frontend:
repository: uugai/hub-frontend
pullPolicy: IfNotPresent
tag: "v1.0.35"
tag: "v1.0.39"
replicas: 1
schema: "http"
url: "yourdomain.com"
@@ -347,18 +347,18 @@ kerberoshub:
tag: "1.0.6436406806"
resources:
requests:
memory: 10Mi
cpu: 10m
memory: 50Mi
cpu: 50m
monitordevice:
repository: kerberos/hub-monitor-device
pullPolicy: IfNotPresent
tag: "1.0.6256190978"
resources:
requests:
memory: 100Mi
memory: 50Mi
cpu: 100m
limits:
memory: 100Mi
memory: 50Mi
cpu: 100m
# E-mail templates
#volumeMounts:
@@ -374,11 +374,11 @@ kerberoshub:
tag: "1.0.6256184527"
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
forwarder:
enabled: false
#repository: kerberos/vault-forwarder
@@ -395,11 +395,11 @@ kerberoshub:
tag: "1.0.2733724812"
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
kerberospipeline:
event:
repository: kerberos/pipe-event
@@ -408,11 +408,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
monitor:
repository: uugai/hub-pipeline-monitor
pullPolicy: IfNotPresent
@@ -420,11 +420,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
sequence:
repository: uugai/hub-pipeline-sequence
pullPolicy: IfNotPresent
@@ -432,11 +432,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
throttler:
repository: kerberos/pipe-throttler
pullPolicy: IfNotPresent
@@ -444,11 +444,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
notify:
repository: uugai/hub-pipeline-notification
pullPolicy: IfNotPresent
@@ -456,11 +456,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
# E-mail templates
#volumeMounts:
# - name: custom-email-templates
@@ -476,11 +476,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
# E-mail templates
#volumeMounts:
# - name: custom-email-templates
@@ -496,11 +496,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
memory: 50Mi
cpu: 50m
dominantColor:
repository: kerberos/pipe-dominantcolor
pullPolicy: IfNotPresent
@@ -508,11 +508,11 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 512Mi
cpu: 250m
memory: 50Mi
cpu: 50m
limits:
memory: 512Mi
cpu: 250m
memory: 50Mi
cpu: 50m
thumbnail:
repository: kerberos/pipe-thumbnail
pullPolicy: IfNotPresent
@@ -525,11 +525,11 @@ kerberospipeline:
enabled: true # If you want to use Kerberos Vault to store the thumbnails
resources:
requests:
memory: 512Mi
cpu: 250m
memory: 50Mi
cpu: 50m
limits:
memory: 512Mi
cpu: 250m
memory: 50Mi
cpu: 50m
counting:
repository: kerberos/pipe-counting
pullPolicy: IfNotPresent
@@ -537,8 +537,8 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 10Mi
cpu: 10m
memory: 50Mi
cpu: 50m
sprite:
enabled: true # Enable or disable the sprite generation 'true' or 'false
repository: uugai/hub-pipeline-sprite
@@ -550,11 +550,11 @@ kerberospipeline:
height: "135" # Should not be changed for the moment (hard coded in UI)
resources:
requests:
memory: 512Mi
cpu: 250m
memory: 50Mi
cpu: 50m
limits:
memory: 512Mi
cpu: 250m
memory: 50Mi
cpu: 50m
export:
repository: kerberos/pipe-export
pullPolicy: IfNotPresent
@@ -562,5 +562,5 @@ kerberospipeline:
replicas: 1
resources:
requests:
memory: 256Mi
cpu: 100m
memory: 50Mi
cpu: 50m

View File

@@ -1,2 +1,4 @@
namespace: minio-tenant
resources:
- minio-tenant-base.yaml
- minio-tenant-base.yaml
- minio-nodeports.yaml

View File

@@ -0,0 +1,17 @@
---
apiVersion: v1
kind: Service
metadata:
name: minio-nodeport
labels:
app: myminio
spec:
type: NodePort
ports:
- port: 9000
targetPort: 9000
nodePort: 30090 # You can specify a port in the range 30000-32767 or let Kubernetes assign one automatically
name: myminio
protocol: TCP
selector:
v1.min.io/tenant: myminio

View File

@@ -1,13 +1,13 @@
#apiVersion: v1
#kind: Namespace
#metadata:
# name: minio-tenant
#---
apiVersion: v1
kind: Namespace
metadata:
name: minio-tenant
---
apiVersion: v1
kind: Secret
metadata:
name: storage-configuration
# namespace: minio-tenant
namespace: minio-tenant
stringData:
config.env: |-
export MINIO_ROOT_USER="minio"
@@ -29,77 +29,190 @@ type: Opaque
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
name: myminio
namespace: minio-tenant
## Optionally pass labels to be applied to the statefulset pods
labels:
app: minio
## Optionally pass annotations to be applied to the statefulset pods
annotations:
prometheus.io/path: /minio/v2/metrics/cluster
prometheus.io/port: "9000"
prometheus.io/scrape: "true"
labels:
app: minio
name: myminio
# namespace: minio-tenant
## If a scheduler is specified here, Tenant pods will be dispatched by specified scheduler.
## If not specified, the Tenant pods will be dispatched by default scheduler.
# scheduler:
# name: my-custom-scheduler
spec:
certConfig: {}
features:
## Enable S3 specific features such as Bucket DNS which would allow `buckets` to be
## accessible as DNS entries of form `<bucketname>.minio.namespace.svc.cluster.local`
## This feature is turned off by default
bucketDNS: false
## Specify a list of domains used to access MinIO and Console
domains: { }
## Enable access via SFTP
## This feature is turned off by default
# enableSFTP: false
## Create users in the Tenant using this field. Make sure to create secrets per user added here.
## Secret should follow the format used in `minio-creds-secret`.
users:
- name: storage-user
## Create buckets using the console user
buckets:
- name: "mybucket"
## This field is used only when "requestAutoCert" is set to true. Use this field to set CommonName
## for the auto-generated certificate. Internal DNS name for the pod will be used if CommonName is
## not provided. DNS name format is *.minio.default.svc.cluster.local
certConfig: { }
## PodManagement policy for MinIO Tenant Pods. Can be "OrderedReady" or "Parallel"
## Refer https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
## for details.
podManagementPolicy: Parallel
## Secret with credentials and configurations to be used by MinIO Tenant.
configuration:
name: storage-configuration
env: []
externalCaCertSecret: []
externalCertSecret: []
externalClientCertSecrets: []
features:
bucketDNS: false
domains: {}
image: quay.io/minio/minio:RELEASE.2024-08-03T04-33-23Z
imagePullSecret: {}
## Add environment variables to be set in MinIO container (https://github.com/minio/minio/tree/master/docs/config)
env: [ ]
## serviceMetadata allows passing additional labels and annotations to MinIO and Console specific
## services created by the operator.
serviceMetadata:
minioServiceLabels: { }
minioServiceAnnotations: { }
consoleServiceLabels: { }
consoleServiceAnnotations: { }
## PriorityClassName indicates the Pod priority and hence importance of a Pod relative to other Pods.
## This is applied to MinIO pods only.
## Refer Kubernetes documentation for details https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass/
priorityClassName: ""
## Use this field to provide one or more external CA certificates. This is used by MinIO
## to verify TLS connections with other applications.
## Certificate secret files will be mounted under /tmp/certs/CAs folder, supported types:
## Opaque | kubernetes.io/tls | cert-manager.io/v1alpha2 | cert-manager.io/v1
##
## ie:
##
## externalCaCertSecret:
## - name: ca-certificate-1
## type: Opaque
## - name: ca-certificate-2
## type: Opaque
## - name: ca-certificate-3
## type: Opaque
##
## Create secrets as explained here:
## https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret
externalCaCertSecret: [ ]
## Use this field to provide one or more Secrets with external certificates. This can be used to configure
## TLS for MinIO Tenant pods.
## Certificate secret files will be mounted under /tmp/certs folder, supported types:
## Opaque | kubernetes.io/tls | cert-manager.io/v1alpha2 | cert-manager.io/v1
##
## ie:
##
## externalCertSecret:
## - name: domain-certificate-1
## type: kubernetes.io/tls
## - name: domain-certificate-2
## type: kubernetes.io/tls
## - name:domain-certificate-3
## type: kubernetes.io/tls
##
## Create secrets as explained here:
## https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret
externalCertSecret: [ ]
## Create secrets as explained here:
## https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret
externalClientCertSecrets: [ ]
## Registry location and Tag to download MinIO Server image
image: quay.io/minio/minio:RELEASE.2024-11-07T00-52-20Z
imagePullSecret: { }
## Mount path where PV will be mounted inside container(s).
mountPath: /export
podManagementPolicy: Parallel
## Sub path inside Mount path where MinIO stores data.
## WARNING:
## We recommend you to keep the same mountPath and the same subPath once the
## Tenant has been deployed over your different PVs.
## This is because if you change these values once Tenant is deployed, then
## you will end up with multiple paths for different buckets. So please, be
## very careful to keep same value for the life of the Tenant.
subPath: ""
## Service account to be used for all the MinIO Pods
serviceAccountName: ""
## Specification for MinIO Pool(s) in this Tenant.
pools:
- affinity:
nodeAffinity: {}
podAffinity: {}
podAntiAffinity: {}
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
## Servers specifies the number of MinIO Tenant Pods / Servers in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 1
## custom name for the pool
name: pool-0
nodeSelector: {}
resources: {}
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
servers: 1
tolerations: []
topologySpreadConstraints: []
## Specify one or more Pod Topology Spread Constraints to apply to pods deployed in the MinIO pool.
## https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints
topologySpreadConstraints: [ ]
## volumesPerServer specifies the number of volumes attached per MinIO Tenant Pod / Server.
volumesPerServer: 4
## nodeSelector parameters for MinIO Pods. It specifies a map of key-value pairs. For the pod to be
## eligible to run on a node, the node must have each of the
## indicated key-value pairs as labels.
## Read more here: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: { }
## Used to specify a toleration for a pod
tolerations: [ ]
## Affinity settings for MinIO pods. Read more about affinity
## here: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity.
affinity:
nodeAffinity: { }
podAffinity: { }
podAntiAffinity: { }
## Configure resource requests and limits for MinIO containers
resources: { }
## This VolumeClaimTemplate is used across all the volumes provisioned for MinIO Tenant in this
## Pool.
volumeClaimTemplate:
apiVersion: v1
kind: persistentvolumeclaims
metadata: {}
metadata: { }
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Ti
storageClassName: openebs-hostpath
status: {}
volumesPerServer: 4
priorityClassName: ""
storageClassName: "ssd-hostpath"
status: { }
## Configure Pod's security context
## We recommend to skip the recursive permission change by using
## fsGroupChangePolicy as OnRootMismatch because it can be pretty
## expensive for larger volumes with lots of small files.
securityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
fsGroup: 1000
fsGroupChangePolicy: "OnRootMismatch"
## Configure container security context
containerSecurityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
## Enable automatic Kubernetes based certificate generation and signing as explained in
## https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster
requestAutoCert: false
serviceAccountName: ""
serviceMetadata:
consoleServiceAnnotations: {}
consoleServiceLabels: {}
minioServiceAnnotations: {}
minioServiceLabels: {}
subPath: ""
users:
- name: storage-user
# The minimum number of days to expiry before an alert for an expiring certificate is fired.
# In the below example, if a given certificate will expire in 7 days then expiration events will only be triggered 1 day before expiry
# certExpiryAlertThreshold: 1
## Prometheus Operator's Service Monitor for MinIO Tenant Pods.
# prometheusOperator:
# labels:
# app: minio-sm
## Audit Logs will be deprecated soon, commenting out for now!.

View File

@@ -27,11 +27,11 @@ spec:
image: uugai/vault:v1.0.14
resources:
requests:
memory: 512Mi
cpu: 256m
memory: 128Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 256m
memory: 128Mi
cpu: 100m
ports:
- containerPort: 80
envFrom:

View File

@@ -47,9 +47,4 @@ helmCharts:
# For some reason it doesn't work with a namespace
#namespace: vernemq
repo: https://vernemq.github.io/docker-vernemq
valuesFile: ../../base/vernemq/vernemq-values.yaml
- name: minio-operator
releaseName: minio-operator
# For some reason it doesn't work with a namespace
#namespace: minio
repo: https://operator.min.io
valuesFile: ../../base/vernemq/vernemq-values.yaml