Analysis Engine

Overview

The exfil-analysis crate (analysis/) is a pure library crate that contains all detection logic for ExfilGuardian. It is split into two sides:

Agent-side: lightweight pre-processing modules that run on the endpoint
Server-side: heavy analysis modules (DPI, signatures, behavioral, correlation) that run on the central server

analysis/src/
├── lib.rs
├── config.rs                          # AnalysisConfig
├── error.rs                           # ExfilError (thiserror)
├── types/
│   ├── packet.rs                      # PacketMetadata, TransportProtocol, Direction
│   ├── flow.rs                        # FlowKey, FlowRecord, PreFilterFlag
│   ├── detection.rs                   # Detection, Confidence, Severity, ThreatCategory
│   ├── alert.rs                       # Alert, AlertStatus, ActionRecommendation
│   ├── dns.rs                         # DnsMetadata
│   └── tls.rs                         # TlsMetadata
├── agent/
│   ├── flow_tracker.rs                # Flow aggregation (5-tuple hashing)
│   ├── dns_extractor.rs               # DNS metadata extraction from port 53
│   ├── tls_extractor.rs               # TLS ClientHello / JA3 extraction
│   └── pre_filter.rs                  # Heuristic flagging for server prioritization
└── server/
    ├── pipeline.rs                    # Orchestrator: fan-out to analyzers, fan-in to correlator
    ├── dpi/
    │   ├── dns_analyzer.rs            # Shannon entropy, DGA scoring, tunnel detection
    │   ├── tls_analyzer.rs            # JA3 matching, certificate anomaly detection
    │   └── icmp_analyzer.rs           # Covert channel detection, payload size analysis
    ├── signatures/
    │   ├── rule.rs                    # SignatureRule struct
    │   ├── loader.rs                  # YAML hot-reload (file watcher)
    │   ├── condition.rs               # Condition evaluator (field matching, regex, thresholds)
    │   └── engine.rs                  # Rule matching engine
    ├── behavioral/
    │   ├── baseline.rs                # Welford online algorithm for running mean/variance
    │   ├── anomaly.rs                 # Z-score anomaly detection, CUSUM change-point
    │   ├── beaconing.rs               # Coefficient of variation for interval regularity
    │   └── volume.rs                  # EMA (Exponential Moving Average) volume tracking
    └── correlation/
        ├── temporal.rs                # Sliding time window for grouping detections
        ├── scorer.rs                  # Bayesian confidence combination
        └── engine.rs                  # Alert generation and lifecycle management

Core Types

PacketMetadata

The atomic unit sent by the agent to the server. Represents a single intercepted packet's metadata, enriched with process context (an endpoint agent advantage over pure NIDS):

pub struct PacketMetadata {
    pub agent_id: Uuid,
    pub timestamp_ns: u64,
    pub flow_id: u64,
    pub src_ip: IpAddr,
    pub dst_ip: IpAddr,
    pub src_port: u16,
    pub dst_port: u16,
    pub protocol: TransportProtocol,
    pub packet_size: u32,
    pub direction: Direction,

    // Endpoint agent context
    pub process_id: u32,
    pub process_name: String,
    pub process_path: Option<String>,

    // Pre-extracted protocol metadata
    pub dns: Option<DnsMetadata>,
    pub tls: Option<TlsMetadata>,

    // Network flags
    pub tcp_flags: Option<u8>,
    pub ttl: Option<u8>,
    pub icmp_type: Option<u8>,
    pub icmp_code: Option<u8>,
    pub icmp_payload_size: Option<u16>,
}

FlowRecord

Aggregated flow produced by the agent's flow_tracker. Reduces gRPC volume by grouping packets by 5-tuple:

pub struct FlowRecord {
    pub key: FlowKey,           // (agent_id, src_ip, dst_ip, src_port, dst_port, protocol)
    pub flow_id: u64,
    pub process_name: String,
    pub process_id: u32,
    pub first_seen_ns: u64,
    pub last_seen_ns: u64,

    // Counters
    pub packet_count: u64,
    pub byte_count: u64,
    pub outbound_packets: u64,
    pub outbound_bytes: u64,
    pub inbound_packets: u64,
    pub inbound_bytes: u64,

    // Derived statistics
    pub avg_packet_size: f64,
    pub avg_interval_ms: f64,
    pub interval_stddev_ms: f64,    // Low = regular = potential C2 beaconing
    pub outbound_ratio: f64,        // High (>0.8) = exfiltration suspicion

    // Pre-filter flags
    pub pre_filter_flags: Vec<PreFilterFlag>,
}

Detection

Individual detection produced by any analyzer (DPI, signatures, behavioral):

pub struct Detection {
    pub detector: DetectorId,       // Which analyzer produced this
    pub category: ThreatCategory,   // What threat was identified
    pub confidence: Confidence,     // 0.0 to 1.0
    pub severity: Severity,         // Low | Medium | High | Critical
    pub description: String,
    pub indicators: Vec<Indicator>, // IOCs (domains, IPs, JA3 hashes, Z-scores...)
    pub agent_id: Uuid,
    pub flow_id: u64,
    pub timestamp_ns: u64,
}

Alert

Final correlated alert presented to the operator. Aggregates multiple Detection values:

pub struct Alert {
    pub id: Uuid,
    pub category: ThreatCategory,
    pub confidence: Confidence,     // Bayesian combination of all detections
    pub severity: Severity,         // May be escalated by correlation
    pub status: AlertStatus,        // New | Investigating | Confirmed | FalsePositive | Resolved

    pub agent_ids: Vec<Uuid>,       // Multi-agent correlation
    pub source_ips: Vec<IpAddr>,
    pub destination_ips: Vec<IpAddr>,
    pub processes: Vec<String>,
    pub flow_ids: Vec<u64>,

    pub detections: Vec<Detection>, // All underlying detections
    pub indicators: Vec<Indicator>,
    pub summary: String,
    pub recommendation: ActionRecommendation,
}

Confidence

Score in the range [0.0, 1.0] with two key operations:

impl Confidence {
    /// Bayesian combination: P(A or B) = 1 - (1-P(A)) * (1-P(B))
    pub fn combine(self, other: Self) -> Self;

    /// Temporal decay: older events contribute less
    pub fn with_decay(self, age_seconds: f64, half_life_seconds: f64) -> Self;
}

These modules run on the endpoint inside the agent process. They are designed to be lightweight and avoid false-positive generation; their purpose is to reduce volume and prioritize traffic for the server.

flow_tracker

Aggregates raw packets into FlowRecord values by hashing the 5-tuple (src_ip, dst_ip, src_port, dst_port, protocol). Computes running statistics (mean packet size, interval standard deviation, outbound ratio) using Welford's algorithm.

dns_extractor

Parses DNS queries and responses from port 53 traffic. Extracts DnsMetadata (query name, record type, response IPs, TTL) and attaches it to the PacketMetadata.

tls_extractor

Parses TLS ClientHello messages to extract SNI (Server Name Indication) and computes JA3 fingerprints. Attaches TlsMetadata to the PacketMetadata.

pre_filter

Applies fast heuristic rules to flag suspicious flows before they reach the server. Flags include:

Flag	Heuristic
`HighOutboundVolume`	Outbound bytes exceed threshold for flow duration
`RegularIntervals`	Low coefficient of variation in packet intervals
`UnusualProcess`	Process name not in expected network-access list
`NewDestination`	Destination IP not seen in local history
`HighDnsFrequency`	DNS query rate exceeds baseline
`LargeIcmpPayload`	ICMP payload size exceeds normal range
`TlsToRawIp`	TLS connection to a raw IP (no domain)

Server-Side Modules

DPI (Deep Packet Inspection)

dns_analyzer

Detects DNS-based exfiltration and C2 communication:

Shannon entropy of subdomain labels: high entropy indicates encoded data (base32/base64)
DGA scoring: algorithmic domain detection via character frequency analysis
Tunnel detection: identifies DNS tunneling tools (iodine, dnscat2, DNSExfiltrator)
Query frequency analysis: abnormal query rates to a single domain

Threat categories: DnsTunneling, DnsExfiltration, DgaDomain, DnsFlood

tls_analyzer

Detects TLS anomalies:

JA3 fingerprint matching against a database of known-malicious fingerprints
Certificate anomaly detection: self-signed certs, SNI mismatch, unusual validity periods
TLS to raw IP: connections using TLS without a domain name

Threat categories: TlsMaliciousFingerprint, TlsCertAnomaly, TlsToRawIp

icmp_analyzer

Detects ICMP-based covert channels:

Payload size analysis: ICMP echo payloads larger than the standard 64 bytes
Pattern detection: structured data in ICMP payloads (ptunnel, icmptunnel signatures)

Threat categories: IcmpCovertChannel, IcmpLargePayload

Signatures

YAML-based detection rules with hot-reload support.

rule.rs

Defines the SignatureRule structure: rule ID, name, severity, conditions, and metadata.

loader.rs

Watches the signatures/ directory for YAML file changes and reloads rules at runtime without restarting the server.

condition.rs

Evaluates rule conditions against packet/flow metadata. Supports field matching, regex patterns, numeric thresholds, and boolean operators (AND, OR, NOT).

engine.rs

Iterates over loaded rules and matches them against incoming FlowRecord and PacketMetadata values. Produces Detection with DetectorId::Signature { rule_id }.

Behavioral Analysis

Statistical anomaly detection based on learned baselines.

baseline.rs

Implements Welford's online algorithm for computing running mean and variance without storing all historical data:

\bar{x}_n = \bar{x}_{n-1} + \frac{x_n - \bar{x}_{n-1}}{n}

M_{2,n} = M_{2,n-1} + (x_n - \bar{x}_{n-1})(x_n - \bar{x}_n)

\sigma^2_n = \frac{M_{2,n}}{n - 1}

Baselines are maintained per-agent, per-metric (bytes/sec, packets/sec, DNS query rate, etc.).

anomaly.rs

Detects deviations from baseline using:

Z-score: $z = \frac{x - \mu}{\sigma}$ , flags values beyond configurable thresholds (default: 3.0)
CUSUM (Cumulative Sum): detects sustained shifts that Z-score might miss on individual samples

beaconing.rs

Detects C2 beaconing patterns by computing the coefficient of variation (CV) of inter-packet intervals:

CV = \frac{\sigma(\Delta t)}{\mu(\Delta t)}

A low CV ( $< 0.15$ ) indicates highly regular communication, characteristic of automated C2 callbacks.

volume.rs

Tracks data volume using an Exponential Moving Average (EMA) to detect gradual or sudden exfiltration:

\text{EMA}(t) = \alpha \cdot x(t) + (1 - \alpha) \cdot \text{EMA}(t-1)

Fires when current volume exceeds $\text{EMA} + k \cdot \sigma$ .

P(\text{threat}) = 1 - \prod_{i} \big(1 - P(d_i)\big)

Applies temporal decay so older detections contribute less to the final score.

engine.rs

The correlation engine that:

Receives Detection values from all analyzers
Groups them by flow/agent/destination via the temporal window
Computes aggregate confidence via the Bayesian scorer
Escalates severity when multiple detection types converge
Emits Alert values with lifecycle status and action recommendations

Pipeline Orchestration

The analysis pipeline on the server follows a fan-out / fan-in pattern:

DPI, signatures, and behavioral analysis run in parallel on each incoming flow. Their Detection outputs are fed into the correlation engine, which produces final Alert values.

Testing

The crate includes 18 unit tests covering:

Confidence combination and temporal decay math
Severity ordering and weight calculations
FlowRecord derived statistics (duration, throughput, packet rate)
Alert multi-agent detection
PreFilterFlag generation

Run the test suite:

cargo test -p exfil-analysis

Analysis Engine

Overview

Core Types

PacketMetadata

FlowRecord

Detection

Alert

Confidence

Agent-Side Modules

flow_tracker

dns_extractor

tls_extractor

pre_filter

Server-Side Modules

DPI (Deep Packet Inspection)

dns_analyzer

tls_analyzer

icmp_analyzer

Signatures

rule.rs

loader.rs

condition.rs

engine.rs

Behavioral Analysis

baseline.rs

anomaly.rs

beaconing.rs

volume.rs

Correlation

temporal.rs

scorer.rs

engine.rs

Pipeline Orchestration

Testing

On this page