Algorithms

L2: Shannon Entropy

Shannon entropy measures the randomness of a string. It is the primary signal used to detect DNS tunneling and DGA domains: legitimate domain names have low entropy (human-readable words), while encoded payloads have high entropy.

Formula

Where $p(x_i)$ is the probability of each character appearing in the string.

Interpretation for DNS

Reference implementation (Rust)

pub fn shannon_entropy(s: &str) -> f64 {
    let len = s.len() as f64;
    let mut freq = [0u32; 256];
    for b in s.bytes() {
        freq[b as usize] += 1;
    }
    freq.iter()
        .filter(|&&c| c > 0)
        .map(|&c| {
            let p = c as f64 / len;
            -p * p.log2()
        })
        .sum()
}

Sources

• Shannon, C.E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal.

L2: N-gram Language Model (DGA Detection)

Domain Generation Algorithms (DGA) produce pseudo-random domain names that look nothing like human language. An n-gram model trained on legitimate domains scores a query domain against known linguistic patterns; low score = likely DGA.

Approach

1. Build a bigram/trigram frequency table from a corpus of legitimate domains (Alexa Top 1M)
2. Score each query: $\text{score} = \prod_{i} P(\text{gram}_i)$
3. Low log-probability → DGA candidate

Thresholds

A log-probability below -15 on trigrams is commonly used as a DGA threshold in production systems (Bambenek, 2015).

Sources

• Antonakakis, M. et al. (2012). From Throw-Away Traffic to Bots: Detecting the Rise of DGA-Based Malware. USENIX Security.
• Plohmann, D. et al. (2016). A Comprehensive Measurement Study of Domain Generating Malware. USENIX Security.

L3: Condition Evaluator

The signature engine evaluates AND-combined conditions against parsed packet fields. Each condition is a triple (field, operator, value).

Operator set

Performance note: regex patterns are compiled once when the YAML rule is loaded and cached. Never compile a regex inside the hot path.

L4: Z-score Anomaly Detection

The behavioral layer establishes a baseline (mean and standard deviation) for traffic metrics over a learning window, then flags deviations that exceed a configured sigma threshold.

Formula

• $x$: observed metric value (packets/s, bytes/s, DNS queries/min...)
• $\mu$: rolling mean over the baseline window
• $\sigma$: rolling standard deviation

A |z| > 3 is the standard 3-sigma rule (99.7% of normal traffic falls within ±3σ).

Sources

• Grubbs, F.E. (1969). Procedures for Detecting Outlying Observations in Samples. Technometrics.

L4: CUSUM (Cumulative Sum Control Chart)

Z-score reacts to point anomalies. CUSUM detects sustained trends: a slow, gradual increase in DNS query rate that individually never crosses the z-score threshold but collectively represents an exfiltration campaign.

Formula

• $\mu_0$: target mean (baseline)
• $k$: slack parameter (typically $\sigma/2$)
• Alert when $S_n > h$ (decision threshold, typically $5\sigma$)

Why both Z-score and CUSUM?

Sources

• Page, E.S. (1954). Continuous Inspection Schemes. Biometrika.
• Basseville, M. & Nikiforov, I. (1993). Detection of Abrupt Changes. Prentice-Hall.

L4: Exponential Moving Average (Baseline)

EMA is used to maintain a lightweight, continuously-updated traffic baseline without storing a full history window.

Formula

• $\alpha$: smoothing factor (0.1–0.3 typical for slow-moving baselines)
• Low $\alpha$ → slow to adapt, resistant to sudden spikes
• High $\alpha$ → adapts quickly, more reactive

Implementation

pub struct EmaBaseline {
    value: f64,
    alpha: f64,
}

impl EmaBaseline {
    pub fn update(&mut self, sample: f64) -> f64 {
        self.value = self.alpha * sample + (1.0 - self.alpha) * self.value;
        self.value
    }
}

L2: JA3 / JA3S Fingerprinting (TLS)

JA3 produces an MD5 hash from specific fields of the TLS ClientHello message. The same client software (and the same malware family) always produces the same JA3 hash, regardless of destination. This allows detection of known malicious TLS clients without decrypting any content.

Fields hashed (ClientHello)

TLS version + Cipher suites + Extensions + Elliptic curves + EC point formats

All concatenated and MD5-hashed → 32-character hex fingerprint.

JA3S is the server-side equivalent: hashes ServerHello fields to fingerprint the server's TLS stack.

Example

Sources

• Althouse, J., Atkins, J., Atkinson, J. (2017). TLS Fingerprinting with JA3 and JA3S. Salesforce Engineering Blog.
• github.com/salesforce/ja3

L5: Confidence Score Aggregation

The correlation engine aggregates evidence from L3 and L4 into a single confidence score using a weighted combination, then applies temporal decay to avoid stale evidence inflating scores.

Aggregation

• $e_i$: evidence score from each layer event (0.0–1.0)
• $w_i$: weight for that layer (L3 signature match = 0.6, L4 anomaly = 0.4 by default)
• Capped at 1.0

Temporal decay

Evidence older than the sliding window (default: 60 seconds) is discounted:

• $\lambda$: decay rate (configurable)
• $\Delta t$: age of the event in seconds

Severity mapping