S4E On-Prem provides comprehensive observability through metrics, logs, and health endpoints. This guide covers setting up monitoring infrastructure and configuring log collection for your deployment.

Observability Stack

S4E On-Prem is designed to integrate with standard Kubernetes observability tools:

Prometheus Metrics

Service Metrics

All S4E services expose Prometheus-compatible metrics at the /metrics endpoint. Key metrics include:

s4e-core

Workers (scan, crawler, dispatcher)

Infrastructure

Prometheus Configuration

Add a ServiceMonitor resource for each S4E service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: s4e-core-monitor
  namespace: s4e
spec:
  selector:
    matchLabels:
      app: s4e-core
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

Grafana Dashboards

S4E provides pre-built Grafana dashboard JSON files covering:

• Platform Overview -- service health, request rates, error rates.
• Scan Activity -- scan throughput, queue depths, worker utilization.
• Infrastructure -- database connections, Redis memory, RabbitMQ message rates.
• Resource Usage -- CPU, memory, and storage consumption per service.

Import the dashboard JSON files from the S4E release artifacts or configure them through ArgoCD.

Logging

Log Format

All S4E services emit structured JSON logs:

{
  "timestamp": "2025-01-15T10:23:45.123Z",
  "level": "INFO",
  "service": "s4e-core",
  "module": "auth",
  "message": "User login successful",
  "user_id": 42,
  "ip": "10.0.1.15",
  "request_id": "abc-123-def"
}

Log Levels

Configure the log level per service via the LOG_LEVEL environment variable.

ELK Stack Integration

Filebeat DaemonSet

Deploy Filebeat as a DaemonSet to collect container logs from all Kubernetes nodes:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    spec:
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:8.12.0
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: containers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: containers
          hostPath:
            path: /var/lib/docker/containers

Kibana Index Patterns

Create index patterns in Kibana for S4E logs:

• s4e-core-* -- API and authentication logs
• s4e-scan-* -- Scan execution logs
• s4e-crawler-* -- Crawl pipeline logs
• s4e-* -- All S4E service logs combined

Fluentd / Fluent Bit Alternative

If you use Fluent Bit instead of the ELK stack, configure a filter to parse S4E JSON logs:

[FILTER]
    Name         parser
    Match        kube.s4e-*
    Key_Name     log
    Parser       json

[OUTPUT]
    Name         es
    Match        kube.s4e-*
    Host         elasticsearch.logging.svc
    Port         9200
    Index        s4e-logs

Health Checks

Liveness and Readiness Probes

All S4E services expose health endpoints used by Kubernetes probes:

The readiness probe verifies connectivity to required dependencies (database, Redis, RabbitMQ) before marking the pod as ready.

Monitoring Health

Create alerts for health check failures:

groups:
  - name: s4e-health
    rules:
      - alert: S4EServiceDown
        expr: up{namespace="s4e"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "S4E service {{ $labels.job }} is down"

Best Practices

1. Retain logs for compliance -- configure log retention policies that meet your regulatory requirements (typically 90-365 days).
2. Use request IDs -- the request_id field enables end-to-end request tracing across services.
3. Monitor queue depths -- rising RabbitMQ queue depths indicate worker capacity issues.
4. Set up PagerDuty or OpsGenie -- route critical alerts to your on-call rotation.
5. Dashboard rotation -- display the Platform Overview dashboard on a wall monitor in your operations center.