Metrics Cardinality Reference

Overview

Budget target: < 5,000 operator-side series for a 50-engine cluster. Gauges with _info suffix follow the info-metric pattern (constant value=1, identity labels only); numeric observations are separate gauges. Never use _total suffix on gauges (reserved for counters by Prometheus convention).

Control-Plane Metrics (operator /metrics on :8443)

Cache Server (existing — internal/rulesets/cache/metrics.go)

Controller Observability (PR #397 — internal/controller/metrics.go)

Note: The metrics in this section are defined in internal/controller/metrics.go (merged via PR #397).

Controller-Runtime Built-ins (not coraza_ prefixed)

These metrics are emitted automatically by controller-runtime and are not configurable by this chart. They appear in the same scrape job as the coraza_ metrics.

Many more standard operator observability metrics are emitted. Refer to the controller-runtime documentation for the full list.

Worked Example: 50-engine cluster

50 Engines × (1 info + 4 conditions)                              =  250 Engine series
50 Engines grouped in 5 namespaces: 5 namespace aggregates         =    5 series
50 RuleSets × (1 info + 3 conditions + 1 sources + 1 data_files)  =  300 RuleSet series
50 RuleSets in 5 namespaces: 5 namespace aggregates                =    5 series
200 RuleSources × (1 info + 2 conditions)                          =  600 RuleSource series
200 RuleSources in 5 namespaces: 5 namespace aggregates            =    5 series
50 referenced RuleData × (1 info + 1 condition)                    =  100 RuleData series
50 RuleData in 5 namespaces: 5 namespace aggregates                =    5 series
Validation counters: 5 ns × (3 rulesource + 2 ruleset outcomes)    =   25 series
Validation histograms: 5 ns × (2×13 rulesource + 2×13 ruleset + 10 cache) =  310 series
Cache server: ~99 series (incl. auth failures counter)
                                                                   ─────────────────
Total coraza_* operator series:                                   ~1704

Well within the 5,000 series budget.

What NOT to use as label values

Rule IDs on the operator side

The operator never sees per-rule decisions — those live in Envoy after the WAF driver processes traffic. Adding rule_id labels on the operator side would require data-plane scraping (see Data-Plane Metrics below). Rule-level cardinality explodes quickly: CRS alone contains thousands of rules.

IP addresses

IP addresses are unbounded cardinality and violate standard Prometheus cardinality policy. Never use client IPs, source IPs, or any IP-derived value as a label.

Numeric counts as label values

Use separate gauge metrics rather than encoding counts in label values. For example, coraza_ruleset_sources{namespace="...", name="..."} carrying the numeric count is correct. A label like source_count="5" on a parent metric is not — each distinct count creates a new time series and the label conveys no structural identity.

Raw error messages

Use error type or reason codes, never free-form error strings. Free-form strings have effectively unbounded cardinality (every unique message is a new series) and are difficult to query reliably.

Data-Plane Metrics (coraza_waf_* — emitted by WAF driver in Envoy)

Data-plane metrics are emitted directly from the Coraza WASM driver running inside Envoy sidecars. They are NOT scraped from the operator.

For the full specification of data-plane metric names, labels, and cardinality constraints (including the top-N rule limit that bounds per-rule series), see driver metrics contract.

Key differences from operator metrics:

ServiceMonitor label handling

The Helm chart enables honorLabels: true on the operator ServiceMonitor so the namespace label on coraza_* metrics reflects the CR namespace, not the operator pod namespace. Prometheus normally overwrites namespace with the scrape target namespace when honorLabels is false.

honorLabels also preserves any job or instance labels emitted by the operator. Those labels are reserved for scrape-target identity; if the operator ever emitted them, aggregation would silently break. The chart therefore applies a built-in metricRelabelings rule to drop job and instance before user-supplied metrics.serviceMonitor.metricRelabelings are applied.

Contract: operator metrics must not emit job or instance labels.

Reducing Cardinality with metricRelabelings

The Helm chart exposes metrics.serviceMonitor.metricRelabelings to drop or transform series before they are stored in your TSDB. The following examples can be pasted into values.yaml.

Example 1 — drop all info metrics in large multi-tenant clusters (keep only conditions + counts):

metrics:
  serviceMonitor:
    metricRelabelings:
      - sourceLabels: [__name__]
        regex: 'coraza_(engine|ruleset|rulesource|ruledata)_info'
        action: drop

This reduces series by 1 per Engine, 1 per RuleSet, 1 per RuleSource, and 1 per referenced RuleData. Useful when you have many short-lived resources and do not need the identity labels captured in info metrics.

Example 2 — keep only coraza_ prefixed metrics (drop controller-runtime built-ins from this scrape job):

metrics:
  serviceMonitor:
    metricRelabelings:
      - sourceLabels: [__name__]
        regex: 'coraza_.*'
        action: keep

This is useful when controller-runtime built-ins are already collected by a cluster-wide scrape job and you want to avoid duplication.

Example 3 — drop high-cardinality cache histogram (keep only counter + gauge):

metrics:
  serviceMonitor:
    metricRelabelings:
      - sourceLabels: [__name__]
        regex: 'coraza_cache_server_request_duration_seconds.*'
        action: drop

The histogram generates ~84 series (6 label combinations × 14 series each: n explicit buckets + 1 +Inf bucket + _count + _sum). Dropping it reduces operator-side series by ~84 while retaining the counter for request rate calculations.