ThreatForge: An ATT&CK ALIGNED Framework for Adversary Emulation and Detection Validation in smart home iot
IoT Security · Adversary Emulation · MITRE ATT&CK · Detection Engineering · Security Research
Cameras, smart locks, bulbs, gateways and companion hosts communicate through protocols and interfaces that can differ significantly from conventional enterprise endpoints. Most adversary-emulation platforms — CALDERA, Atomic Red Team, Metta — are built for enterprise Windows and Linux infrastructure. There’s no reproducible equivalent for consumer IoT and smart-home environments, even though IoT devices are one of the fastest-growing, least-instrumented attack surfaces security teams have to defend. A penetration test can show a smart lock or camera can be compromised. A detection engineer can separately write a rule intended to catch it. What’s usually missing is proof that the two actually connect.
Can We Execute an IoT Attack and Prove That the Security Stack Detected It? A successful attack demonstrates that a security control can be bypassed. For IoT environments, this distinction becomes especially important:
What teams actually need to know:
-
- Was the behavior visible to the security stack?
-
- Was it detected — and how quickly?
-
- What telemetry did it generate?
-
- Can the result be reproduced run-to-run?
-
- Can the detection be improved and re-tested?
ThreatForge is built around a simple objective, Do not assume that a detection works. Exercise the behaviour and measure what the security stack actually observes. This principle led to the development of ThreatForge.
What Is ThreatForge?
ThreatForge is an open framework for IoT adversary emulation and closed-loop detection validation. It connects controlled adversary execution to target-side activity, telemetry and security detection using MITRE CALDERA, a protocol-aware ThreatForge Agent, Node-RED and Wazuh.
Its current architecture integrates:
| Component | Role |
|---|---|
| MITRE CALDERA | ATT&CK-aligned adversary emulation and operation orchestration |
| ThreatForge Agent | CALDERA-compatible agent with native HTTP, MQTT and SSH executors |
| Node-RED | Controlled smart-home IoT simulation |
| Wazuh | Telemetry collection, MITRE-mapped detection and alert validation |
| Docker-Compose | Reproducible isolated deployment |
The framework is designed so that an adversary action can be traced through the environment and into the detection layer rather than being evaluated as an isolated attack event.
The Workflow
From Adversary Action to Measured Detection, ThreatForge experiment follows six stages.
-
- Orchestrate — CALDERA launches the defined adversary profile
-
- Execute — The ThreatForge Agent receives and dispatches task through executors (HTTP / MQTT / SSH)
-
- Generate Activity — The Node-RED testbed produces observable device behavior
-
- Capture Telemetry — Target-side events and logs record what happened
-
- Detect — Wazuh ingests the telemetry and applies its detection rules
-
- Validate — Execution records, device activity, and detection events are correlated to score the result
This creates a complete chain: What the attacker did → what the target generated → what the security stack saw → what the security stack detected.
Threatforge Architecture
| Hop | Protocol / Port | Description |
| CALDERA → ThreatForge Agent | HTTPS 8888 |
Operation tasking, ability retrieval, result reporting |
| Agent → (HTTP) | HTTP 1880 |
Device state queries, snapshot retrieval, REST-style control |
| Agent → (MQTT) | MQTT 1883 |
Topic-based command injection and telemetry subscription |
| Agent → (SSH) | SSH 22 |
Remote command execution against emulated embedded-Linux devices |
| Node-RED → Wazuh | Syslog/Filebeat 514/5044 |
IoT event and log forwarding into the detection pipeline |
| Analyst → Wazuh Dashboard | HTTPS 443 |
Alert triage, rule tuning, detection validation |
ThreatForge separates the agent, CALDERA abilities, adversary profiles, testbed, Wazuh configuration, Docker deployment and research artefacts into distinct components. That modular structure makes the framework suitable for extending individual parts without redesigning the entire experiment.
the Threatforge agent
A Protocol Bridge, Not Just Another Host Agent, Unlike standard host-based CALDERA agents, the ThreatForge Agent acts as a protocol bridge between CALDERA’s tasking model and IoT device interfaces:
-
- Beacon-based tasking — polls CALDERA over HTTPS for queued abilities
-
- Multi-protocol executors — routes each ability to the correct handler (
http_executor,mqtt_executor,ssh_executor)
- Multi-protocol executors — routes each ability to the correct handler (
-
- Fact-driven targeting — consumes CALDERA facts (device IP, MQTT topic, credentials) so abilities generalize across devices instead of being hardcoded
-
- Structured result reporting — returns status, stdout/stderr, and timing back to CALDERA for scoring
-
- Extensible executor interface — new protocols (CoAP, BLE) drop in as a single new executor class
ATT&CK-Aligned IoT Adversary Emulation
Each ability in abilities/ is explicitly tagged with a MITRE ATT&CK technique ID, so an operation produces a measurable, citable coverage map — not an anecdotal attack narrative. The current evaluated profile contains five abilities:
| Evaluated Behaviour | ATT&CK Technique |
|---|---|
| Device and inventory discovery | T1595 — Active Scanning |
| Camera snapshot retrieval | T1125 — Video Capture |
| Smart-lock interaction | T0836 — Modify Parameter |
| Smart-bulb manipulation | T0831 — Manipulation of Control (ICS) |
| SSH-based lateral movement | T1021.004 — Remote Services: SSH |
This enables a consistent mapping from:
Attack behaviour → expected activity → detection rule → validation result
Detection Isn’t an Afterthought — It’s Part of the Experiment
ThreatForge treats detection validation as a core component of adversary emulation.
Wazuh receives the resulting telemetry and applies MITRE-mapped detection rules. The current repository maps the evaluated behaviours to dedicated rule identifiers, including rules for device discovery, camera activity, smart-lock manipulation, smart-bulb manipulation and SSH-based lateral movement. This makes detection engineering measurable rather than purely declarative.
ThreatForge was evaluated across 25 repeated laboratory runs and 125 technique instances. The framework correlates evidence from adversary execution, target activity and detection events using timestamps and a defined 240-second validation window per run. Reported Results:
Detection Coverage – 94.4%, Precision – 89.8%, F1-score – 91.6%, Mean Time to Detect – 59.0 s, False Positives per Run – 0.60
The repository reports these results as evidence that the current implementation can repeatedly execute the evaluated attack scenarios and measure the response of the configured detection stack.
Why This Approach Matters
ThreatForge addresses a specific operational problem: Security controls need to be validated against behaviour, not just configured on paper. For an IoT security team, that can mean asking:
-
- Was the device discovered?
-
- Was the camera activity visible?
-
- Was the MQTT manipulation observed?
-
- Was the smart-lock interaction detected?
-
- Was the lateral-movement activity identified?
-
- How quickly did the detection appear?
-
- Can the same validation be repeated after the rule is changed?
By connecting adversary execution with target-side telemetry and SIEM alerts, ThreatForge provides a controlled workflow for investigating those questions.
Built to Fill a Real Gap
ThreatForge extends established adversary-emulation workflows into a controlled IoT environment, combining ATT&CK-aligned operations with protocol-aware execution, a smart-home testbed and detection validation.
| Capability | Vanilla CALDERA | Atomic Red Team | IoT Honeypots (e.g. IoTPOT) | ThreatForge |
|---|---|---|---|---|
| ATT&CK-aligned operations |  |
|
 |
|
| IoT-native protocol execution (MQTT / HTTP / SSH) | |
|
Partial | |
| Reproducible, containerised smart-home testbed | |
|
|
|
| Detection-rule-to-technique traceability | |
|
|
|
| Passive attacker observation | |
|
|
|
| Purpose-built for smart-home research | |
|
Partial | |
ThreatForge is built on CALDERA and extends it with an IoT-focused agent, protocol layer and reproducible testbed for adversary emulation and detection validation.
Built for the People Who Have to Prove Security Works
| Audience | What They Get |
|---|---|
| Security Researchers | Reproducible and citable IoT adversary-emulation methodology |
| Red & Purple Teams | Controlled adversarial behaviours and defensive-visibility validation |
| Detection Engineers | A way to test whether detection rules trigger against defined behaviours |
| SOC Analysts | Visibility, alert generation and detection-latency investigation |
| IoT Security Practitioners | A device-oriented attack and detection sandbox |
| Academic Institutions & Students | A reproducible environment for research and training |
| Industry / Enterprise Partners | A framework for validating detection-stack assumptions |
| Sponsors | An open research programme that can be expanded through infrastructure, validation and development support |
Quick Start
1. Clone the Repository
git clone https://github.com/giridharan-veda/ThreatForge.git cd ThreatForge
2. Install Dependencies
pip install -r requirements.txt
3. Run the Setup Script
chmod +x scripts/setup.sh ./scripts/setup.sh
4. Launch the Lab
docker compose up -d
5. Open the Security Consoles
| Service | Purpose |
|---|---|
| MITRE CALDERA | Operation control and adversary management |
| Node-RED | IoT testbed visualisation |
| Wazuh Dashboard | Detection monitoring and alerting |
6. Run the Adversary Profile
Launch the defined adversary profile and monitor the resulting activity and detections through the Wazuh dashboard.
System Requirements
| Requirement | Minimum | Recommended |
|---|---|---|
| Operating System | Linux / macOS / Windows (WSL2) | Ubuntu 22.04 LTS |
| CPU | 4 cores | 8 cores |
| RAM | 8 GB | 16 GB |
| Disk | 20 GB free | 40 GB free SSD |
| Docker | 24.x | Latest stable |
| Python | 3.10 | 3.11+ |
What’s Inside the Repository
| Path | Contents |
|---|---|
paper/ |
Research paper, architecture, methodology and evaluation |
docs/ |
Setup guide, usage guide and documentation index |
agent/ |
ThreatForge Agent source |
abilities/ |
MITRE CALDERA abilities |
adversary/ |
Adversary profiles |
testbed/ |
Node-RED smart-home IoT simulation |
wazuh/ |
Detection rules, decoders and dashboards |
docker/ |
Docker and container configuration |
scripts/ |
Setup and helper scripts |
reports/ |
Experimental results referenced by run ID |
CITATION.cff |
Citation metadata for academic use |
The Next Phase: Wider Protocols, More Devices, Bigger Campaigns
The following capabilities represent planned areas of development:
| Planned Area | Direction |
|---|---|
| Additional IoT Device Profiles | Expand the range of simulated smart-home devices and behaviours |
| BLE Support | Add Bluetooth Low Energy attack and validation scenarios |
| Zigbee Support | Extend protocol coverage for connected-device environments |
| ICS / SCADA Simulation | Expand the framework into industrial control scenarios |
| AI-Assisted Adversary Planning | Investigate automated planning and adaptive adversary workflows |
| Automated Detection Validation | Automate attack execution, telemetry correlation and validation |
| Multi-Agent Adversary Emulation | Support coordinated activity across multiple agents |
| Expanded ATT&CK Coverage | Increase the range of evaluated techniques and attack chains |
ThreatForge Is Open. Help Forge What’s Next.
ThreatForge is open to research, engineering and industry collaboration.
| Partner Type | Opportunity |
|---|---|
| Researchers | New IoT attack scenarios, ATT&CK mappings, datasets and evaluation methods |
| Security Engineers | Extend protocol support, agent functionality and detection integrations |
| SOC & Detection Teams | Evaluate the framework against monitoring and detection assumptions |
| Industry Partners | Integrations, joint research and enterprise-oriented validation |
| Sponsors | Support physical-device research, infrastructure, validation campaigns and open development |
Contributing
ThreatForge welcomes contributions to the framework, documentation, detections, adversary profiles and supporting research.
Contribution Workflow
1. Fork the repository
2. Create a feature branch
git checkout -b feature/your-feature
3. Commit your changes
git commit -m “Add your change”
4. Open a Pull Request
For significant changes, open an Issue first to discuss the proposed direction.
GitHub Repository: https://github.com/giridharan-veda/ThreatForge
Trust, License & Citation
| License | MIT License. View the LICENSE → |
| Citation | Cite ThreatForge using the repository’s CITATION.cff metadata. View CITATION.cff → |
| Research Use | Suitable for academic and security research workflows |
| Source | Publicly available through GitHub |
Responsible Use
ThreatForge is intended for educational, research and authorised security-testing purposes.
Users are responsible for complying with applicable laws, regulations and organisational policies.
Final CTA, The Next Experiment Can Be Yours:
-
- Researchers can challenge it.
-
- Developers can extend it.
-
- Security teams can evaluate it.
-
- Industry partners can integrate it.
-
- Sponsors can help expand it.
EXPLORE THREATFORGE ON GITHUB →
ThreatForge
Forging Adversaries. Strengthening Defenders.
MIT Licensed · Open Source · Built on MITRE CALDERA, Wazuh, Node-RED and Docker
Overview · Architecture · ATT&CK Coverage · Quick Start · Documentation · Roadmap · License · Citation · GitHub