Cluster Infrastructure Services

Ominis Cluster Manager uses a two-tier infrastructure model: cluster-wide services (shared) and tenant services (isolated). This document provides a deep dive into the cluster infrastructure layer that powers all tenant deployments.

Introduction

Cluster vs Tenant Infrastructure Model

The Ominis platform uses a two-tier infrastructure approach to balance efficiency, isolation, and operational simplicity:

Cluster Infrastructure (this document):

Deployed once per Kubernetes cluster
Shared by all tenants
Namespaces: cert-manager, authentik, vaultwarden, flow-proxy, homer, excalidraw
Examples: TLS certificate management, identity provider, HTTP ingress, SIP monitoring
Repository: cluster-infra

Tenant Infrastructure:

Deployed per customer/tenant
Isolated in dedicated namespaces
Namespace pattern: client-{tenant-name} (e.g., client-demo-client)
Examples: API servers, queue pods, IVR pods, databases, tenant-specific ingress
Repository: cluster-manager

Why Separate?

This separation provides several key benefits:

Benefit	Description
Cost Efficiency	One cert-manager instance serves all tenants instead of N instances
Operational Simplicity	Single upgrade, single monitoring stack, centralized configuration
Resource Optimization	Shared ingress controller reduces pod overhead
Faster Onboarding	New tenants start immediately without waiting for infrastructure
Security Isolation	Tenant workloads isolated via namespaces and network policies
Centralized Management	Identity, secrets, TLS certificates managed in one place

Relationship to Tenant Infrastructure

This documentation focuses on cluster-level services. For tenant-specific infrastructure (API servers, queues, databases), see Helm Infrastructure Deployment.

Cluster Architecture Overview

The cluster infrastructure consists of six core services that work together to provide shared capabilities:

Service Dependencies

Understanding service dependencies is critical for deployment order and troubleshooting:

Key Dependencies:

Cert-Manager is the foundation (no dependencies)
Authentik requires cert-manager for TLS
Vaultwarden requires cert-manager for TLS
Flow-Proxy discovers ingress resources dynamically
Homer and Excalidraw are independent (optional)

Service Catalog

1. Cert-Manager: Automated TLS Certificate Management

Purpose: Automated TLS certificate management using Let's Encrypt ACME protocol.

Architecture

Cert-Manager uses Kubernetes Custom Resource Definitions (CRDs) to automate certificate lifecycle:

ClusterIssuer: Cluster-wide certificate authority (Let's Encrypt)
Certificate: Defines a certificate request
CertificateRequest: Generated automatically by Certificate
Challenge: ACME HTTP-01 or DNS-01 validation

Certificate Lifecycle

Configuration

ClusterIssuer Definition:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # Let's Encrypt production server
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@ominis.ai
    # Private key for ACME account
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    # HTTP-01 challenge using Traefik ingress
    - http01:
        ingress:
          class: traefik

Certificate Request Example:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: demo-client-api-tls
  namespace: client-demo-client
spec:
  secretName: demo-client-api-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - demo-client-api.app.ominis.ai

Integration with Tenant Ingress

Tenants consume cert-manager via ingress annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: client-demo-client
  annotations:
    # This annotation triggers cert-manager
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - demo-client-api.app.ominis.ai
    secretName: demo-client-api-tls  # Cert-manager creates this
  rules:
  - host: demo-client-api.app.ominis.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api
            port:
              number: 8000

Use Cases

Use Case	Description
Tenant API Ingress	`demo-client-api.app.ominis.ai` automatically gets TLS
Documentation Site	`docs.app.ominis.ai` with Let's Encrypt certificate
Internal Services	Any service with ingress gets free TLS
Automatic Renewal	Certificates renewed 60 days before expiry

Troubleshooting

Certificate not issued:

# Check certificate status
kubectl describe certificate demo-client-api-tls -n client-demo-client

# Check challenges
kubectl get challenges -n client-demo-client

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager --tail=100

Common issues:

DNS not configured: Ensure DNS points to cluster ingress IP
HTTP-01 challenge failed: Check Traefik routing and firewall
Rate limit exceeded: Let's Encrypt has rate limits (50 certs/week per domain)

2. Authentik: Identity and Access Management

Purpose: Enterprise-grade identity provider for SSO, OAuth, OIDC, LDAP, and SAML.

Architecture

Authentik is a flows-based authentication system:

PostgreSQL Backend: Stores users, groups, applications
Flows: Customizable authentication pipelines
Providers: OAuth2/OIDC, SAML, LDAP
Policies: Fine-grained access control
Applications: Integrated services

Features

Feature	Description
Single Sign-On	Users log in once, access multiple services
OAuth2/OIDC Provider	Standard protocol for API authentication
LDAP Server	Bridge to legacy systems
SAML Provider	Enterprise SSO integration
User Management	Self-service enrollment, password reset
Multi-Factor Auth	TOTP, WebAuthn, SMS

Configuration Example

Helm Values:

authentik:
  # PostgreSQL backend
  postgresql:
    enabled: true
    persistence:
      enabled: true
      size: 10Gi
  
  # Ingress configuration
  ingress:
    enabled: true
    ingressClassName: traefik
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    hosts:
      - host: auth.ominis.ai
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: auth-ominis-ai-tls
        hosts:
          - auth.ominis.ai
  
  # Secret key for session encryption
  secret_key: "changeme-random-secret-key"
  
  # Email configuration
  email:
    host: smtp.sendgrid.net
    port: 587
    username: apikey
    password: "SG.xxx"
    from: noreply@ominis.ai

Application Integration

Adding an Application:

Create OAuth2/OIDC Provider in Authentik UI
Configure redirect URIs
Get client ID and secret
Configure application to use Authentik

Example: Grafana Integration:

[auth.generic_oauth]
enabled = true
name = Authentik
client_id = grafana-client-id
client_secret = your-client-secret
scopes = openid profile email
auth_url = https://auth.ominis.ai/application/o/authorize/
token_url = https://auth.ominis.ai/application/o/token/
api_url = https://auth.ominis.ai/application/o/userinfo/

Use Cases

Use Case	Description
Internal Tool SSO	Grafana, Homer UI, admin dashboards
API OAuth	Secure API endpoints with OAuth2
LDAP Bridge	Connect legacy systems requiring LDAP
Multi-Tenant Isolation	User groups per tenant

3. Vaultwarden: Password Management

Purpose: Self-hosted password manager compatible with Bitwarden clients.

Architecture

Rust-based: Lightweight, efficient alternative to official Bitwarden
SQLite/PostgreSQL: Flexible storage backends
Client Compatible: Works with all Bitwarden clients (browser, mobile, CLI)
API Compatible: REST API for automation
Organizations: Team password sharing

Terraform Provider Integration

Vaultwarden includes a custom Terraform provider for infrastructure-as-code:

Provider Configuration:

terraform {
  required_providers {
    vaultwarden = {
      source = "ominis-ai/vaultwarden"
      version = "~> 1.0"
    }
  }
}

provider "vaultwarden" {
  endpoint = "https://vault.ominis.ai"
  email    = "admin@ominis.ai"
  password = var.admin_password
}

Manage Organizations:

resource "vaultwarden_organization" "platform" {
  name = "Platform Team"
}

resource "vaultwarden_organization_collection" "credentials" {
  organization_id = vaultwarden_organization.platform.id
  name           = "Production Credentials"
}

resource "vaultwarden_login_item" "database" {
  organization_id = vaultwarden_organization.platform.id
  collection_id   = vaultwarden_organization_collection.credentials.id
  name            = "PostgreSQL Root"
  username        = "postgres"
  password        = random_password.db_password.result
  uris            = ["postgres://postgres.prod.svc.cluster.local:5432"]
  notes           = "Production database root credentials"
}

Deployment Configuration

Helm Values:

vaultwarden:
  # Image
  image:
    repository: vaultwarden/server
    tag: latest
  
  # Persistence
  persistence:
    enabled: true
    size: 1Gi
  
  # Ingress
  ingress:
    enabled: true
    className: traefik
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    hosts:
      - host: vault.ominis.ai
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: vault-ominis-ai-tls
        hosts:
          - vault.ominis.ai
  
  # Environment variables
  env:
    DOMAIN: "https://vault.ominis.ai"
    SIGNUPS_ALLOWED: "false"
    INVITATIONS_ALLOWED: "true"
    ADMIN_TOKEN: "changeme-admin-token"

Use Cases

Use Case	Description
Team Credentials	Share API keys, passwords securely
API Key Management	Store tokens for external services
Certificate Storage	TLS certificates and private keys
SSH Key Management	Store SSH keys for server access
Infrastructure Secrets	Terraform-managed secret lifecycle

Backup Strategy

Database Backup:

# Backup Vaultwarden data
kubectl exec -n vaultwarden vaultwarden-0 -- \
  tar -czf - /data > vaultwarden-backup-$(date +%Y%m%d).tar.gz

# Upload to S3
aws s3 cp vaultwarden-backup-$(date +%Y%m%d).tar.gz \
  s3://backups/vaultwarden/

4. Flow-Proxy: HTTP Reverse Proxy (Traefik)

Purpose: Dynamic HTTP reverse proxy and Kubernetes ingress controller.

Architecture

Traefik is a cloud-native ingress controller:

Dynamic Configuration: Auto-discovers Kubernetes ingress resources
Middleware System: Composable request/response transformations
TLS Termination: Integrates with cert-manager
Load Balancing: Round-robin, weighted, sticky sessions
Observability: Metrics, tracing, access logs

Ingress Pattern

Standard Tenant Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: client-example
  annotations:
    # Cert-manager integration
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    # Traefik middleware (optional)
    traefik.ingress.kubernetes.io/router.middlewares: "default-security-headers@kubernetescrd"
spec:
  # Traefik ingress class
  ingressClassName: traefik
  tls:
  - hosts:
    - example-api.app.ominis.ai
    secretName: example-api-tls
  rules:
  - host: example-api.app.ominis.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api
            port:
              number: 8000

Middleware System

Traefik middleware enables request/response transformations:

Security Headers Middleware:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: security-headers
  namespace: default
spec:
  headers:
    customResponseHeaders:
      X-Frame-Options: "SAMEORIGIN"
      X-Content-Type-Options: "nosniff"
      X-XSS-Protection: "1; mode=block"
      Referrer-Policy: "no-referrer-when-downgrade"
      Permissions-Policy: "geolocation=(), microphone=(), camera=()"

Rate Limiting Middleware:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: rate-limit
  namespace: default
spec:
  rateLimit:
    average: 100
    burst: 50
    period: 1m

Compression Middleware:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: compress
  namespace: default
spec:
  compress: {}

Apply Middleware to Ingress:

metadata:
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: >-
      default-security-headers@kubernetescrd,
      default-rate-limit@kubernetescrd,
      default-compress@kubernetescrd

Configuration

Helm Values:

flow-proxy:
  # Deployment
  deployment:
    replicas: 2
  
  # Service (LoadBalancer for external access)
  service:
    type: LoadBalancer
    annotations:
      # OVH LoadBalancer
      service.beta.kubernetes.io/ovh-loadbalancer-flavor: "small"
  
  # Ports
  ports:
    web:
      port: 80
      exposedPort: 80
    websecure:
      port: 443
      exposedPort: 443
      tls:
        enabled: true
  
  # TLS options
  tlsOptions:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  
  # Access logs
  logs:
    access:
      enabled: true
      format: json
  
  # Metrics
  metrics:
    prometheus:
      enabled: true

Use Cases

Use Case	Description
Tenant API Routing	Route `{tenant}-api.app.ominis.ai` to tenant pods
Static Site Hosting	Serve documentation, landing pages
WebSocket Proxying	Real-time connections for IVR, monitoring
Path-Based Routing	`/api` → API service, `/docs` → docs service

Dashboard Access

# Port-forward to dashboard
kubectl port-forward -n flow-proxy \
  svc/traefik-dashboard 9000:9000

# Access dashboard
open http://localhost:9000/dashboard/

5. Homer: SIP Capture and VoIP Monitoring

Purpose: SIP protocol capture, analysis, and monitoring using HEP (Homer Encapsulation Protocol).

Architecture

Homer provides end-to-end VoIP monitoring:

Heplify: Capture agent (DaemonSet on all nodes)
Homer-App: Web UI for call flow analysis
PostgreSQL: Timeseries storage for SIP messages
HEP Protocol: Encapsulated SIP packet transport

Components

Heplify DaemonSet

Heplify runs on every node to capture SIP traffic:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: heplify
  namespace: homer
spec:
  selector:
    matchLabels:
      app: heplify
  template:
    metadata:
      labels:
        app: heplify
    spec:
      # Host network to capture traffic
      hostNetwork: true
      hostPID: true
      containers:
      - name: heplify
        image: sipcapture/heplify:latest
        securityContext:
          privileged: true
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
        args:
        # Capture on all interfaces
        - -i
        - any
        # Homer server endpoint
        - -hs
        - homer-app.homer.svc.cluster.local:9060
        # Capture SIP protocol
        - -m
        - SIP
        # Port range for SIP
        - -pr
        - "5060-5061"
        # Capture SIP and SDP
        - -dim
        - sip,sdp
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "500m"

FreeSWITCH HEP Configuration

Configure FreeSWITCH to send SIP traces to Homer:

sofia.conf.xml:

<configuration name="sofia.conf" description="sofia Endpoint">
  <global_settings>
    <!-- Enable HEP capture -->
    <param name="capture-server" value="udp:homer-app.homer.svc.cluster.local:9060"/>
  </global_settings>
  
  <profiles>
    <profile name="internal">
      <!-- Profile-specific HEP -->
      <settings>
        <param name="sip-capture" value="yes"/>
        <param name="capture-server" value="udp:homer-app.homer.svc.cluster.local:9060"/>
      </settings>
    </profile>
  </profiles>
</configuration>

Analysis Features

Feature	Description
Call Flow Diagrams	Ladder diagrams showing SIP message flow
Search & Filter	Find calls by caller ID, method, response code
Statistics	Call success rate, average duration, error rates
Real-Time Monitoring	Live SIP traffic analysis
Export	PCAP export for Wireshark analysis

Use Cases

Use Case	Description
Call Troubleshooting	Debug failed calls with full SIP trace
Quality Monitoring	Track call success rates, errors
Regulatory Compliance	SIP message logging for audits
Performance Analysis	Identify bottlenecks in call flow

6. Excalidraw: Collaborative Whiteboarding

Purpose: Real-time collaborative whiteboard for architecture diagrams and brainstorming.

Architecture

React-based: Modern web application
Real-Time Collaboration: WebSocket-based multi-user editing
Export Formats: PNG, SVG, JSON
Embedding: Diagrams can be embedded in documentation
Privacy: Self-hosted, data stays in cluster

Deployment Configuration

Helm Values:

excalidraw:
  # Deployment
  replicas: 2
  image:
    repository: excalidraw/excalidraw
    tag: latest
  
  # Persistence for saved diagrams
  persistence:
    enabled: true
    size: 5Gi
    storageClass: ""
  
  # Ingress
  ingress:
    enabled: true
    className: traefik
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    hosts:
      - host: draw.ominis.ai
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: draw-ominis-ai-tls
        hosts:
          - draw.ominis.ai
  
  # Resources
  resources:
    requests:
      memory: "128Mi"
      cpu: "100m"
    limits:
      memory: "256Mi"
      cpu: "200m"

Use Cases

Use Case	Description
Architecture Diagrams	Network topology, system design
Incident Documentation	Draw current vs desired state
Team Brainstorming	Real-time collaborative design sessions
Documentation Assets	Export diagrams for docs

Deployment Order

Services must be deployed in order due to dependencies:

Rationale:

Cert-Manager first: Required for TLS certificates
Authentik second: Needs cert-manager for TLS
Vaultwarden third: Needs cert-manager for TLS
Flow-Proxy fourth: Discovers ingress resources after they exist
Homer & Excalidraw: Independent, can be deployed in parallel

Deployment Sequence

Deployment Procedures

Prerequisites Check

Before deployment, verify cluster readiness:

# Verify Kubernetes access
kubectl cluster-info

# Check Kubernetes version (1.24+)
kubectl version --short

# Verify Helm installation
helm version

# Check storage class exists
kubectl get storageclass

# Verify DNS is configured
dig app.ominis.ai

# Check LoadBalancer support (for flow-proxy)
kubectl get svc -A | grep LoadBalancer

Step 1: Deploy Cert-Manager

Cert-Manager is the foundation for TLS automation:

# Add Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Install cert-manager with CRDs
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.0 \
  --set installCRDs=true

# Wait for readiness
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/instance=cert-manager \
  -n cert-manager \
  --timeout=300s

# Verify installation
kubectl get pods -n cert-manager

Create ClusterIssuer:

# Create Let's Encrypt production issuer
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@ominis.ai
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: traefik
EOF

# Verify ClusterIssuer
kubectl get clusterissuer letsencrypt-prod

Step 2: Deploy Authentik

cd /home/matt/projects/fml/cluster-infra

# Install Authentik
helm install authentik helm-charts/authentik/ \
  --namespace authentik \
  --create-namespace \
  -f helm-charts/authentik/values.yaml

# Wait for pods
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=authentik \
  -n authentik \
  --timeout=300s

# Verify deployment
kubectl get pods -n authentik
kubectl get certificate -n authentik

# Get bootstrap password
kubectl get secret -n authentik authentik-bootstrap \
  -o jsonpath='{.data.password}' | base64 -d

Access Authentik UI:

# Open browser
open https://auth.ominis.ai

# Default credentials:
# Username: akadmin
# Password: (from bootstrap secret)

Step 3: Deploy Vaultwarden

# Install Vaultwarden
helm install vaultwarden helm-charts/vaultwarden/ \
  --namespace vaultwarden \
  --create-namespace \
  -f helm-charts/vaultwarden/values.yaml

# Wait for readiness
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=vaultwarden \
  -n vaultwarden \
  --timeout=180s

# Get admin token
kubectl get secret -n vaultwarden vaultwarden-admin \
  -o jsonpath='{.data.token}' | base64 -d

# Verify
kubectl get pods -n vaultwarden
kubectl get certificate -n vaultwarden

Access Vaultwarden:

open https://vault.ominis.ai

Step 4: Deploy Flow-Proxy (Traefik)

# Install Traefik
helm install flow-proxy helm-charts/flow-proxy/ \
  --namespace flow-proxy \
  --create-namespace \
  -f helm-charts/flow-proxy/values.yaml

# Wait for LoadBalancer IP
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=traefik \
  -n flow-proxy \
  --timeout=300s

# Get external IP
kubectl get svc -n flow-proxy traefik -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# Verify ingress controller
kubectl get pods -n flow-proxy
kubectl get svc -n flow-proxy

Configure DNS:

# Point wildcard DNS to LoadBalancer IP
# A record: *.app.ominis.ai → <EXTERNAL_IP>

Step 5: Deploy Homer (Optional)

# Install Homer
helm install homer helm-charts/homer/ \
  --namespace homer \
  --create-namespace \
  -f helm-charts/homer/values.yaml

# Deploy Heplify DaemonSet
kubectl apply -f manifests/homer/heplify-daemonset.yaml

# Verify
kubectl get pods -n homer
kubectl get daemonset -n homer heplify

Access Homer UI:

open https://homer.ominis.ai

Step 6: Deploy Excalidraw (Optional)

# Install Excalidraw
helm install excalidraw helm-charts/excalidraw/ \
  --namespace excalidraw \
  --create-namespace \
  -f helm-charts/excalidraw/values.yaml

# Wait for readiness
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=excalidraw \
  -n excalidraw \
  --timeout=180s

# Verify
kubectl get pods -n excalidraw
kubectl get certificate -n excalidraw

Access Excalidraw:

open https://draw.ominis.ai

Verification & Testing

Cert-Manager Verification

Test Certificate Creation:

# Create test certificate
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-cert
  namespace: default
spec:
  secretName: test-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - test.app.ominis.ai
EOF

# Check certificate status
kubectl describe certificate test-cert -n default

# Wait for certificate to be ready
kubectl wait --for=condition=ready certificate test-cert -n default --timeout=120s

# Verify secret created
kubectl get secret test-tls -n default

Authentik Testing

# Access Authentik UI
open https://auth.ominis.ai

# Get bootstrap credentials
kubectl get secret -n authentik authentik-bootstrap \
  -o jsonpath='{.data.password}' | base64 -d

# Test OAuth2 endpoint
curl https://auth.ominis.ai/application/o/.well-known/openid-configuration

Traefik Dashboard

# Port-forward to dashboard
kubectl port-forward -n flow-proxy \
  svc/traefik-dashboard 9000:9000

# Access dashboard
open http://localhost:9000/dashboard/

# Check ingress routes
kubectl get ingressroute -A

Homer Access

# Check Homer UI
open https://homer.ominis.ai

# Check HEP traffic
kubectl logs -n homer -l app=heplify --tail=100

# Check Homer database
kubectl exec -n homer homer-postgres-0 -- \
  psql -U homer -c "SELECT COUNT(*) FROM hep_proto_1_call;"

Operational Procedures

Upgrade Cluster Service

# Update Helm repository
helm repo update

# Preview changes (requires helm-diff plugin)
helm diff upgrade cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.14.0

# Perform upgrade
helm upgrade cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.14.0 \
  --reuse-values

# Verify upgrade
kubectl get pods -n cert-manager
kubectl rollout status deployment -n cert-manager cert-manager

# Rollback if needed
helm rollback cert-manager -n cert-manager

Monitor Service Health

Check All Cluster Services:

# One-liner to check all namespaces
for ns in cert-manager authentik vaultwarden flow-proxy homer excalidraw; do
  echo "=== $ns ==="
  kubectl get pods -n $ns
done

# Check ingress status
kubectl get ingress -A

# Check certificate status
kubectl get certificate -A

# Check PVC usage
kubectl get pvc -A

Check Resource Usage:

# Top pods per namespace
kubectl top pods -n authentik
kubectl top pods -n vaultwarden
kubectl top pods -n flow-proxy

# Check logs
kubectl logs -n cert-manager -l app=cert-manager --tail=50
kubectl logs -n flow-proxy -l app.kubernetes.io/name=traefik --tail=50

Backup and Disaster Recovery

Backup Procedures

Vaultwarden Database:

# Backup PostgreSQL database
kubectl exec -n vaultwarden vaultwarden-postgres-0 -- \
  pg_dump -U vaultwarden vaultwarden | gzip > vaultwarden-backup-$(date +%Y%m%d).sql.gz

# Upload to S3
aws s3 cp vaultwarden-backup-$(date +%Y%m%d).sql.gz \
  s3://ominis-backups/vaultwarden/

# Automated daily backup (CronJob)
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vaultwarden-backup
  namespace: vaultwarden
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:15
            command:
            - /bin/sh
            - -c
            - |
              pg_dump -h vaultwarden-postgres -U vaultwarden vaultwarden | \
              gzip > /backup/vaultwarden-\$(date +%Y%m%d).sql.gz
            volumeMounts:
            - name: backup
              mountPath: /backup
          volumes:
          - name: backup
            persistentVolumeClaim:
              claimName: backup-pvc
          restartPolicy: OnFailure
EOF

Authentik Database:

# Backup Authentik
kubectl exec -n authentik authentik-postgres-0 -- \
  pg_dump -U authentik authentik | gzip > authentik-backup-$(date +%Y%m%d).sql.gz

# Upload to S3
aws s3 cp authentik-backup-$(date +%Y%m%d).sql.gz \
  s3://ominis-backups/authentik/

Certificate Secrets:

# Export all TLS certificates
kubectl get secret -A \
  -l cert-manager.io/certificate-name \
  -o yaml > certificates-backup-$(date +%Y%m%d).yaml

# Upload to S3
aws s3 cp certificates-backup-$(date +%Y%m%d).yaml \
  s3://ominis-backups/certificates/

Disaster Recovery

Restore Cluster Services:

# 1. Deploy cert-manager first
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.0 \
  --set installCRDs=true

# 2. Restore ClusterIssuer
kubectl apply -f manifests/cert-manager/cluster-issuer.yaml

# 3. Deploy Authentik
helm install authentik helm-charts/authentik/ \
  --namespace authentik \
  --create-namespace

# 4. Restore Authentik database
kubectl exec -n authentik authentik-postgres-0 -- \
  psql -U authentik -d authentik < authentik-backup.sql

# 5. Deploy other services
helm install vaultwarden helm-charts/vaultwarden/ --namespace vaultwarden --create-namespace
helm install flow-proxy helm-charts/flow-proxy/ --namespace flow-proxy --create-namespace

# 6. Restore Vaultwarden
kubectl exec -n vaultwarden vaultwarden-postgres-0 -- \
  psql -U vaultwarden -d vaultwarden < vaultwarden-backup.sql

# 7. Verify certificate renewal
kubectl get certificate -A

Troubleshooting

Certificate Not Issued

Symptoms:

Certificate status: False
Ingress not accessible via HTTPS

Debug Steps:

# Check certificate status
kubectl describe certificate <name> -n <namespace>

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager --tail=100

# Check ACME challenges
kubectl get challenges -A

# Describe challenge
kubectl describe challenge <challenge-name> -n <namespace>

Common Issues:

Issue	Solution
DNS not configured	Ensure DNS points to LoadBalancer IP
HTTP-01 challenge failed	Check Traefik routing, firewall rules
Rate limit exceeded	Let's Encrypt: 50 certs/week per domain
Invalid email	Update ClusterIssuer with valid email

Ingress Not Working

Symptoms:

Service not accessible externally
404 errors

Debug Steps:

# Check Traefik logs
kubectl logs -n flow-proxy -l app.kubernetes.io/name=traefik --tail=100

# Verify ingress resource
kubectl describe ingress <name> -n <namespace>

# Check service endpoints
kubectl get endpoints <service> -n <namespace>

# Check pod status
kubectl get pods -n <namespace>

# Port-forward to test service directly
kubectl port-forward -n <namespace> svc/<service> 8080:80
curl http://localhost:8080

Common Issues:

Issue	Solution
Service has no endpoints	Check pod labels match service selector
Wrong port in ingress	Verify service port matches ingress backend
Missing ingress class	Add `ingressClassName: traefik`
DNS not updated	Wait for DNS propagation or check records

Service Unavailable

Symptoms:

Pods not starting
CrashLoopBackOff

Debug Steps:

# Check pod status
kubectl get pods -n <namespace>

# Check pod logs
kubectl logs -n <namespace> <pod-name>

# Check previous pod logs (if restarting)
kubectl logs -n <namespace> <pod-name> --previous

# Describe pod for events
kubectl describe pod <pod-name> -n <namespace>

# Check events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Common Issues:

Issue	Solution
Image pull errors	Check image name, registry credentials
Insufficient resources	Increase node capacity or reduce requests
Volume mount issues	Check PVC exists and is bound
Configuration errors	Review ConfigMaps and Secrets

Homer Not Capturing Traffic

Symptoms:

No SIP calls in Homer UI
Heplify logs show no traffic

Debug Steps:

# Check Heplify DaemonSet
kubectl get daemonset -n homer heplify

# Check Heplify logs
kubectl logs -n homer -l app=heplify --tail=100

# Verify FreeSWITCH HEP configuration
kubectl exec -n <tenant-namespace> <queue-pod> -- \
  fs_cli -x "sofia status profile internal"

# Check Homer database
kubectl exec -n homer homer-postgres-0 -- \
  psql -U homer -c "SELECT COUNT(*) FROM hep_proto_1_call;"

Common Issues:

Issue	Solution
FreeSWITCH not configured	Add `sip-capture=yes` to Sofia profile
Wrong Homer endpoint	Update capture-server parameter
Network policy blocking	Allow traffic from pods to Homer
Heplify not running	Check DaemonSet status, hostNetwork=true

Integration with Tenant Infrastructure

How Tenants Consume Cluster Services

Tenants consume cluster services through standard Kubernetes patterns:

1. TLS Certificates

Tenants request certificates via ingress annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: client-demo-client
  annotations:
    # Reference ClusterIssuer (cluster service)
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: traefik  # Use cluster ingress controller
  tls:
  - hosts:
    - demo-client-api.app.ominis.ai
    secretName: demo-client-api-tls  # Cert-manager creates this
  rules:
  - host: demo-client-api.app.ominis.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api
            port:
              number: 8000

Flow:

Tenant creates Ingress with cert-manager.io/cluster-issuer annotation
Cert-manager detects annotation, creates Certificate resource
Certificate issued via Let's Encrypt ACME challenge
Secret created in tenant namespace with TLS cert/key
Traefik discovers ingress and terminates TLS

2. Ingress Routing

Tenants define Ingress resources, Traefik auto-discovers:

Pattern: {tenant}-api.app.ominis.ai

# Tenant A
host: demo-client-api.app.ominis.ai

# Tenant B
host: acme-corp-api.app.ominis.ai

Traefik Configuration:

Automatically discovers all Ingress resources
Routes based on host field
Applies middleware if specified
Terminates TLS using certificate secret

3. Authentication (Future)

Tenants can integrate with Authentik for OAuth:

Traefik Middleware for OAuth:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: oauth-auth
  namespace: client-demo-client
spec:
  forwardAuth:
    address: https://auth.ominis.ai/outpost.goauthentik.io/auth/traefik
    trustForwardHeader: true
    authResponseHeaders:
      - X-Auth-User
      - X-Auth-Email

Apply to Ingress:

metadata:
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: >-
      client-demo-client-oauth-auth@kubernetescrd

4. SIP Monitoring

Tenant queue pods send SIP traffic to Homer:

FreeSWITCH Configuration (in queue pod):

<configuration name="sofia.conf">
  <global_settings>
    <param name="capture-server" value="udp:homer-app.homer.svc.cluster.local:9060"/>
  </global_settings>
</configuration>

Heplify (DaemonSet on all nodes):

Captures SIP packets on all interfaces
Encapsulates in HEP protocol
Sends to Homer API
Labels traffic by pod name/namespace

Homer UI:

Tenant-scoped call analysis
Search by namespace or pod name
Full SIP trace for troubleshooting

Security Considerations

Network Policies

Tenant Isolation:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cross-tenant
  namespace: client-demo-client
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  # Allow from same namespace
  - from:
    - namespaceSelector:
        matchLabels:
          name: client-demo-client
  # Allow from ingress controller
  - from:
    - namespaceSelector:
        matchLabels:
          name: flow-proxy
  egress:
  # Allow DNS
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
  # Allow to cluster services
  - to:
    - namespaceSelector:
        matchLabels:
          name: cert-manager
  - to:
    - namespaceSelector:
        matchLabels:
          name: homer

RBAC

Tenant Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-service-account
  namespace: client-demo-client
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: api-role
  namespace: client-demo-client
rules:
# Allow managing own resources
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: api-rolebinding
  namespace: client-demo-client
subjects:
- kind: ServiceAccount
  name: api-service-account
  namespace: client-demo-client
roleRef:
  kind: Role
  name: api-role
  apiGroup: rbac.authorization.k8s.io

Restrictions:

Tenants cannot modify ClusterIssuer
Tenants cannot access other namespaces
Tenants cannot modify flow-proxy configuration
Service accounts scoped to tenant namespace

Secrets Management

Best Practices:

Kubernetes secrets encrypted at rest (enable encryption provider)
Vaultwarden for sensitive credentials (API keys, passwords)
Certificate rotation automated (cert-manager)
Secrets never in version control

Example: Store API Key in Vaultwarden:

# Using Bitwarden CLI
bw config server https://vault.ominis.ai
bw login admin@ominis.ai

# Create secure note
bw create item \
  --organizationid <org-id> \
  --collectionid <collection-id> \
  --type secureNote \
  --name "Demo Client API Key" \
  --notes "Winnipeg2025"

TLS Everywhere

Policy: All HTTP traffic must use TLS

Enforcement:

# Traefik redirect HTTP to HTTPS
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: default
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
# Apply globally to Traefik
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: flow-proxy
  namespace: flow-proxy
spec:
  values:
    additionalArguments:
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"

Certificate Rotation:

Let's Encrypt certificates valid 90 days
Cert-manager renews 60 days before expiry
Automatic renewal, no manual intervention

Architecture Decision Records

ADR-001: Cluster-Wide vs Tenant-Scoped Services

Context: Infrastructure services (cert-manager, ingress, identity) can be deployed at two levels:

Cluster-level: One instance shared by all tenants
Tenant-level: Each tenant gets their own instance

Decision: Deploy cert-manager, authentik, vaultwarden, flow-proxy, homer, and excalidraw at cluster-level.

Alternatives Considered:

Approach	Pros	Cons
Tenant-scoped everything	Perfect isolation, tenant control over versions	Massive resource overhead, complex operations
Hybrid (some cluster, some tenant)	Balanced approach	Inconsistent patterns, confusing
Cluster-wide only ✅	Cost efficient, simple operations	Shared failure domain, version lock

Consequences:

✅ Pros:

Cost Efficiency: One cert-manager vs N cert-managers saves ~500MB RAM × N tenants
Operational Simplicity: Single upgrade path, single monitoring stack
Resource Optimization: Shared Traefik reduces pod overhead (2 replicas vs 2×N)
Centralized Management: Identity, secrets, TLS in one place
Faster Onboarding: New tenants start instantly, no infrastructure wait time

❌ Cons:

Shared Failure Domain: Cert-manager failure affects all tenants
Version Lock: All tenants on same service versions
Security Boundary: Requires strict RBAC and network policies
Scaling Limits: Cluster services must handle all tenant load

Mitigation:

High Availability: Run 2+ replicas for critical services (cert-manager, Traefik)
Network Policies: Strict namespace isolation
Resource Quotas: Prevent noisy neighbors
Staged Rollouts: Blue-green deployments for cluster service updates
Monitoring: Alert on cluster service health

Status: Accepted

Date: 2025-10-14

ADR-002: Traefik Over Other Ingress Controllers

Context: Kubernetes requires an ingress controller to route external HTTP traffic to services. Options include:

Nginx Ingress: Most popular, mature, battle-tested
HAProxy: High performance, low-level control
Traefik: Modern, cloud-native, Kubernetes-native
Istio: Service mesh with ingress capabilities

Decision: Use Traefik as the cluster ingress controller (flow-proxy).

Alternatives Considered:

Controller	Pros	Cons
Nginx Ingress	Most popular, mature, extensive docs	Complex config, limited middleware, manual cert-manager setup
HAProxy	Highest performance, granular control	Steep learning curve, less Kubernetes-native
Traefik ✅	Dynamic config, native middleware, cert-manager integration	Moderate performance vs HAProxy
Istio	Full service mesh, advanced features	Overkill for simple routing, complex

Why Traefik?:

Feature	Benefit
Dynamic Configuration	Auto-discovers Kubernetes Ingress resources, no reload
Middleware System	Composable transformations (headers, rate limiting, compression)
Cert-Manager Integration	First-class TLS automation, seamless
Modern Dashboard	Real-time routing visualization
Cloud-Native	Built for containers and microservices
Let's Encrypt Native	ACME protocol built-in

Trade-offs:

Aspect	Traefik	Nginx Ingress
Configuration	Annotations	ConfigMap + Annotations
Performance	Very Good (10K req/s)	Excellent (15K req/s)
Middleware	Native, composable	Limited, via snippets
Dashboard	Yes, built-in	No
Cert-Manager	Seamless	Manual setup
Learning Curve	Moderate	Steep

Consequences:

✅ Benefits:

All ingress resources use ingressClassName: traefik
Middleware pattern for headers, auth, rate limiting
Dashboard available at traefik.ominis.ai (or port-forward)
Annotation-based configuration (simple)
Auto-discovery reduces operational overhead

❌ Drawbacks:

Slightly lower raw performance than Nginx (~30% fewer req/s)
Different annotation namespace (traefik.ingress.kubernetes.io/)
Team needs to learn Traefik patterns

Mitigation:

Use multiple Traefik replicas (2+) for high availability
Cache static assets at CDN layer if performance becomes bottleneck
Document standard middleware patterns for team

Status: Accepted

Date: 2025-10-14

System Overview - High-level Ominis architecture
Helm Infrastructure - Tenant-level deployment with Helm
Queue Management - Tenant queue service example
Testing Strategy - Testing cluster and tenant infrastructure

Summary

Ominis Cluster Manager uses a two-tier infrastructure model for efficiency and isolation:

Cluster Services (this document):

6 services deployed once per cluster
Shared by all tenants
Examples: Cert-Manager (TLS), Traefik (ingress), Authentik (SSO), Vaultwarden (secrets), Homer (SIP monitoring), Excalidraw (collaboration)

Tenant Services:

Deployed per customer
Isolated in namespaces
Examples: API servers, queues, databases

This approach provides:

✅ Cost efficiency (shared resources)
✅ Operational simplicity (centralized management)
✅ Fast tenant onboarding (no infrastructure wait)
✅ Security isolation (network policies, RBAC)

Next Steps:

Review Helm Infrastructure for tenant deployment
Explore Queue Management for tenant service example
Check Testing Strategy for infrastructure testing

Powered by Ominis.ai - Modern telephony infrastructure for the cloud era.

Introduction​

Cluster vs Tenant Infrastructure Model​

Why Separate?​

Cluster Architecture Overview​

Service Dependencies​

Service Catalog​

1. Cert-Manager: Automated TLS Certificate Management​

Architecture​

Certificate Lifecycle​

Configuration​

Integration with Tenant Ingress​

Use Cases​

Troubleshooting​

2. Authentik: Identity and Access Management​

Architecture​

Features​

Configuration Example​

Application Integration​

Use Cases​

3. Vaultwarden: Password Management​

Architecture​

Terraform Provider Integration​

Deployment Configuration​

Use Cases​

Backup Strategy​

4. Flow-Proxy: HTTP Reverse Proxy (Traefik)​

Architecture​

Ingress Pattern​

Middleware System​

Configuration​

Use Cases​

Dashboard Access​

5. Homer: SIP Capture and VoIP Monitoring​

Architecture​

Components​

Heplify DaemonSet​

FreeSWITCH HEP Configuration​

Analysis Features​

Use Cases​

6. Excalidraw: Collaborative Whiteboarding​

Architecture​

Deployment Configuration​

Use Cases​

Deployment Order​

Deployment Sequence​

Deployment Procedures​

Prerequisites Check​

Step 1: Deploy Cert-Manager​

Step 2: Deploy Authentik​

Step 3: Deploy Vaultwarden​

Step 4: Deploy Flow-Proxy (Traefik)​

Step 5: Deploy Homer (Optional)​

Step 6: Deploy Excalidraw (Optional)​

Verification & Testing​

Cert-Manager Verification​

Authentik Testing​

Traefik Dashboard​

Homer Access​

Operational Procedures​

Upgrade Cluster Service​

Monitor Service Health​

Backup and Disaster Recovery​

Backup Procedures​

Disaster Recovery​

Troubleshooting​

Certificate Not Issued​

Ingress Not Working​

Service Unavailable​

Homer Not Capturing Traffic​

Integration with Tenant Infrastructure​

How Tenants Consume Cluster Services​

1. TLS Certificates​

2. Ingress Routing​

3. Authentication (Future)​

4. SIP Monitoring​

Security Considerations​

Network Policies​

RBAC​

Secrets Management​

TLS Everywhere​

Introduction

Cluster vs Tenant Infrastructure Model

Why Separate?

Cluster Architecture Overview

Service Dependencies

Service Catalog

1. Cert-Manager: Automated TLS Certificate Management

Architecture

Certificate Lifecycle

Configuration

Integration with Tenant Ingress

Use Cases

Troubleshooting

2. Authentik: Identity and Access Management

Architecture

Features

Configuration Example

Application Integration

Use Cases

3. Vaultwarden: Password Management

Architecture

Terraform Provider Integration

Deployment Configuration

Use Cases

Backup Strategy

4. Flow-Proxy: HTTP Reverse Proxy (Traefik)

Architecture

Ingress Pattern

Middleware System

Configuration

Use Cases

Dashboard Access

5. Homer: SIP Capture and VoIP Monitoring

Architecture

Components

Heplify DaemonSet

FreeSWITCH HEP Configuration

Analysis Features

Use Cases

6. Excalidraw: Collaborative Whiteboarding

Architecture

Deployment Configuration

Use Cases

Deployment Order

Deployment Sequence

Deployment Procedures

Prerequisites Check

Step 1: Deploy Cert-Manager

Step 2: Deploy Authentik

Step 3: Deploy Vaultwarden

Step 4: Deploy Flow-Proxy (Traefik)

Step 5: Deploy Homer (Optional)

Step 6: Deploy Excalidraw (Optional)

Verification & Testing

Cert-Manager Verification

Authentik Testing

Traefik Dashboard

Homer Access

Operational Procedures

Upgrade Cluster Service

Monitor Service Health

Backup and Disaster Recovery

Backup Procedures

Disaster Recovery

Troubleshooting

Certificate Not Issued

Ingress Not Working

Service Unavailable

Homer Not Capturing Traffic

Integration with Tenant Infrastructure

How Tenants Consume Cluster Services

1. TLS Certificates

2. Ingress Routing

3. Authentication (Future)

4. SIP Monitoring

Security Considerations

Network Policies

RBAC

Secrets Management

TLS Everywhere