Uncategorized

ThreatForge: An ATT&CK ALIGNED Framework for Adversary Emulation and Detection Validation in smart home iot   IoT Security · Adversary Emulation · MITRE ATT&CK · Detection Engineering · Security Research Cameras, smart locks, bulbs, gateways and companion hosts communicate through protocols and interfaces that can differ significantly from conventional enterprise endpoints. Most adversary-emulation platforms — CALDERA, Atomic Red Team, Metta — are built for enterprise Windows and Linux infrastructure. There’s no reproducible equivalent for consumer IoT and smart-home environments, even though IoT devices are one of the fastest-growing, least-instrumented attack surfaces security teams have to defend. A penetration test can show a smart lock or camera can be compromised. A detection engineer can separately write a rule intended to catch it. What’s usually missing is proof that the two actually connect.   Can We Execute an IoT Attack and Prove That the Security Stack Detected It? A successful attack demonstrates that a security control can be bypassed. For IoT environments, this distinction becomes especially important: What teams actually need to know: Was the behavior visible to the security stack? Was it detected — and how quickly? What telemetry did it generate? Can the result be reproduced run-to-run? Can the detection be improved and re-tested? ThreatForge is built around a simple objective, Do not assume that a detection works. Exercise the behaviour and measure what the security stack actually observes. This principle led to the development of ThreatForge. What Is ThreatForge?   ThreatForge is an open framework for IoT adversary emulation and closed-loop detection validation. It connects controlled adversary execution to target-side activity, telemetry and security detection using MITRE CALDERA, a protocol-aware ThreatForge Agent, Node-RED and Wazuh. Its current architecture integrates: Component Role MITRE CALDERA ATT&CK-aligned adversary emulation and operation orchestration ThreatForge Agent CALDERA-compatible agent with native HTTP, MQTT and SSH executors Node-RED Controlled smart-home IoT simulation Wazuh Telemetry collection, MITRE-mapped detection and alert validation Docker-Compose Reproducible isolated deployment The framework is designed so that an adversary action can be traced through the environment and into the detection layer rather than being evaluated as an isolated attack event. The Workflow From Adversary Action to Measured Detection, ThreatForge experiment follows six stages.   Orchestrate — CALDERA launches the defined adversary profile Execute — The ThreatForge Agent receives and dispatches task through executors (HTTP / MQTT / SSH) Generate Activity — The Node-RED testbed produces observable device behavior Capture Telemetry — Target-side events and logs record what happened Detect — Wazuh ingests the telemetry and applies its detection rules Validate — Execution records, device activity, and detection events are correlated to score the result This creates a complete chain: What the attacker did → what the target generated → what the security stack saw → what the security stack detected. Threatforge Architecture   Hop Protocol / Port Description CALDERA → ThreatForge Agent HTTPS 8888 Operation tasking, ability retrieval, result reporting Agent → (HTTP) HTTP 1880 Device state queries, snapshot retrieval, REST-style control Agent → (MQTT) MQTT 1883 Topic-based command injection and telemetry subscription Agent → (SSH) SSH 22 Remote command execution against emulated embedded-Linux devices Node-RED → Wazuh Syslog/Filebeat 514/5044 IoT event and log forwarding into the detection pipeline Analyst → Wazuh Dashboard HTTPS 443 Alert triage, rule tuning, detection validation ThreatForge separates the agent, CALDERA abilities, adversary profiles, testbed, Wazuh configuration, Docker deployment and research artefacts into distinct components. That modular structure makes the framework suitable for extending individual parts without redesigning the entire experiment. the Threatforge agent A Protocol Bridge, Not Just Another Host Agent, Unlike standard host-based CALDERA agents, the ThreatForge Agent acts as a protocol bridge between CALDERA’s tasking model and IoT device interfaces: Beacon-based tasking — polls CALDERA over HTTPS for queued abilities Multi-protocol executors — routes each ability to the correct handler (http_executor, mqtt_executor, ssh_executor) Fact-driven targeting — consumes CALDERA facts (device IP, MQTT topic, credentials) so abilities generalize across devices instead of being hardcoded Structured result reporting — returns status, stdout/stderr, and timing back to CALDERA for scoring Extensible executor interface — new protocols (CoAP, BLE) drop in as a single new executor class ATT&CK-Aligned IoT Adversary Emulation Each ability in abilities/ is explicitly tagged with a MITRE ATT&CK technique ID, so an operation produces a measurable, citable coverage map — not an anecdotal attack narrative. The current evaluated profile contains five abilities:   Evaluated Behaviour ATT&CK Technique Device and inventory discovery T1595 — Active Scanning Camera snapshot retrieval T1125 — Video Capture Smart-lock interaction T0836 — Modify Parameter Smart-bulb manipulation T0831 — Manipulation of Control (ICS) SSH-based lateral movement T1021.004 — Remote Services: SSH This enables a consistent mapping from: Attack behaviour → expected activity → detection rule → validation result   Detection Isn’t an Afterthought — It’s Part of the Experiment ThreatForge treats detection validation as a core component of adversary emulation. Wazuh receives the resulting telemetry and applies MITRE-mapped detection rules. The current repository maps the evaluated behaviours to dedicated rule identifiers, including rules for device discovery, camera activity, smart-lock manipulation, smart-bulb manipulation and SSH-based lateral movement. This makes detection engineering measurable rather than purely declarative.       ThreatForge was evaluated across 25 repeated laboratory runs and 125 technique instances. The framework correlates evidence from adversary execution, target activity and detection events using timestamps and a defined 240-second validation window per run. Reported Results:   Detection Coverage – 94.4%, Precision – 89.8%, F1-score – 91.6%, Mean Time to Detect – 59.0 s, False Positives per Run – 0.60 The repository reports these results as evidence that the current implementation can repeatedly execute the evaluated attack scenarios and measure the response of the configured detection stack.   Why This Approach Matters ThreatForge addresses a specific operational problem: Security controls need to be validated against behaviour, not just configured on paper. For an IoT security team, that can mean asking: Was the device discovered? Was the camera activity visible? Was the MQTT manipulation observed? Was the smart-lock interaction detected? Was the lateral-movement activity identified? How quickly did the detection appear? Can the same validation