S4E On-Prem uses RabbitMQ as the primary message broker for asynchronous inter-service communication. All scan jobs, crawl pipeline stages, and event notifications flow through RabbitMQ queues.

Overview

RabbitMQ serves as the backbone for S4E's event-driven architecture:

• Scan dispatch -- scan requests are published to queues and consumed by worker services.
• Crawl pipeline -- multi-stage crawl operations pass through a chain of queues.
• Event notifications -- triggers and actions are coordinated through message passing.
• Work distribution -- messages are distributed across multiple worker replicas for parallel processing.

Deployment Options

Option 1: In-Cluster RabbitMQ (Helm Subchart)

The S4E Helm chart includes a RabbitMQ subchart:

# s4e-values.yaml
rabbitmq:
  enabled: true
  auth:
    username: s4e_mq
    password: "<strong-password>"
  persistence:
    enabled: true
    size: 20Gi
    storageClass: ssd
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 2000m
      memory: 4Gi

Option 2: External RabbitMQ

Connect to an existing RabbitMQ cluster:

rabbitmq:
  enabled: false

core:
  env:
    RABBITMQ_HOST: "rabbitmq-cluster.messaging.internal"
    RABBITMQ_PORT: "5672"
    RABBITMQ_USER: "s4e_mq"
    RABBITMQ_VHOST: "s4e"
  secrets:
    RABBITMQ_PASS: "<rabbitmq-password>"

Option 3: Clustered RabbitMQ

For high availability, deploy a RabbitMQ cluster with mirrored queues:

rabbitmq:
  enabled: true
  replicaCount: 3
  clustering:
    enabled: true
  auth:
    username: s4e_mq
    password: "<strong-password>"

Queue Architecture

Exchange Topology

S4E uses a topic exchange model:

Queue Definitions

Scan Queues

Crawler Pipeline Queues

Virtual Hosts

For multi-tenant or environment isolation, configure separate vhosts:

rabbitmqctl add_vhost s4e_production
rabbitmqctl add_vhost s4e_staging
rabbitmqctl set_permissions -p s4e_production s4e_mq ".*" ".*" ".*"

Configuration

Consumer Settings

Message Durability

All S4E queues are configured as durable by default:

• Messages are persisted to disk.
• Queues survive broker restarts.
• Consumer acknowledgments ensure at-least-once delivery.

Dead Letter Handling

Messages that fail processing after the configured retry count are routed to the dead letter exchange:

Original Queue --> (reject after N retries) --> s4e.dead_letter --> dead_letter_queue

Monitor the dead_letter_queue for messages that require manual investigation.

Performance Tuning

Prefetch Count

The prefetch count controls how many messages a consumer fetches before processing:

S4E scan workers should use a lower prefetch count because scan operations are long-running. Crawler pipeline stages can use a higher prefetch count for throughput.

Memory and Disk Alarms

Configure RabbitMQ resource limits:

# rabbitmq.conf
vm_memory_high_watermark.relative = 0.6
vm_memory_high_watermark_paging_ratio = 0.5
disk_free_limit.relative = 2.0

When the memory watermark is reached, RabbitMQ stops accepting new messages from publishers, which causes backpressure on S4E services.

Connection Limits

# rabbitmq.conf
channel_max = 2047

Ensure the maximum channel count accommodates all S4E service replicas.

Monitoring

Management UI

RabbitMQ Management UI is available at port 15672:

kubectl -n s4e port-forward svc/rabbitmq 15672:15672

Access at http://localhost:15672 with your configured credentials.

Key Metrics

Prometheus Integration

Enable the RabbitMQ Prometheus plugin:

rabbitmq-plugins enable rabbitmq_prometheus

Metrics are exposed at http://rabbitmq:15692/metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rabbitmq-monitor
  namespace: s4e
spec:
  selector:
    matchLabels:
      app: rabbitmq
  endpoints:
    - port: prometheus
      path: /metrics
      interval: 30s

Troubleshooting

Next Steps

• Queue optimization -- advanced queue tuning for high throughput.
• Worker services -- understand the message consumers.
• Environment variables -- complete configuration reference.