docs: document dlp ioc enrichment pipeline

2026-06-12 03:46:43 +03:00
parent aece5c381f
commit 61a9887ebb
4 changed files with 388 additions and 46 deletions
@@ -1,72 +1,162 @@
 # DLP IOC Enrichment from Hayabusa/Sigma

-This adds a safe offline pipeline to preload DLP blacklists from static Sigma indicators.
+This document describes the automatic DLP IOC/signature replenishment pipeline.
+It is separate from the DLP Policy Engine lifecycle and from the Hayabusa EVTX
+forensics runner.
+
+## Purpose
+
+The pipeline preloads DLP indicator blacklists from static Sigma indicators. It
+is used to enrich endpoint DLP rules without manually editing endpoint JSON
+policy files.

 ## Source

- Sigma rules from Hayabusa ruleset (`hayabusa-rules` YAML files).
+- Upstream ruleset: `Yamato-Security/hayabusa-rules`
+- Default source URL:
+  `https://github.com/Yamato-Security/hayabusa-rules/archive/refs/heads/main.zip`
+- Ansible variable: `aw_dlp_ioc_rules_zip_url`
+- Production enable flag: `aw_dlp_ioc_enabled`

-## Extracted indicators
+Hayabusa rules may carry licenses that are separate from the Hayabusa binary.
+Check the upstream ruleset license before packaging or redistributing generated
+artifacts.
+
+## Production Pipeline
+
+When `aw_dlp_ioc_enabled=true`, `ansible/deploy_aw_server.yml` installs and
+starts this chain on the AW server:
+
+1. `aw-dlp-ioc-refresh.timer` runs on boot and then every
+   `aw_dlp_ioc_refresh_interval` (`6h` by default).
+2. The timer starts `aw-dlp-ioc-refresh.service`.
+3. The service executes `/usr/local/bin/aw-dlp-ioc-refresh.sh`.
+4. The wrapper downloads `aw_dlp_ioc_rules_zip_url`.
+5. The wrapper unpacks `hayabusa-rules` Sigma YAML files into a temporary
+   working directory.
+6. The wrapper runs `/usr/local/bin/aw-extract-ioc-from-sigma`.
+7. The Rust extractor writes generated IOC artifacts to
+   `/opt/activitywatch/dlp-ioc/output`.
+8. `aw-worktime-api` serves the artifacts from `/dlp-ioc/...` for DLP policy
+   consumption.
+
+Production units:
+
+- `aw-dlp-ioc-refresh.service`
+- `aw-dlp-ioc-refresh.timer`
+
+Production paths:
+
+- workdir: `/opt/activitywatch/dlp-ioc`
+- output dir: `/opt/activitywatch/dlp-ioc/output`
+- latest symlink: `/opt/activitywatch/dlp-ioc/latest`
+
+## Extractor
+
+Primary extractor:
+
+- crate: `adk-rust/crates/extract-ioc-from-sigma`
+- installed binary: `/usr/local/bin/aw-extract-ioc-from-sigma`
+- local build:
+
+```bash
+cd <PROJECT_ROOT>/adk-rust
+cargo build --release -p extract-ioc-from-sigma
+```
+
+Local/manual wrapper:
+
+- `scripts/build_dlp_ioc_from_hayabusa.sh`
+
+```bash
+cd <PROJECT_ROOT>
+bash scripts/build_dlp_ioc_from_hayabusa.sh \
+  /mnt/usb_hdd1/Projects/hayabusa/rules \
+  <PROJECT_ROOT>/data/dlp-ioc
+```
+
+The old Python extractor path is not the production path. Do not document
+`scripts/extract_ioc_from_sigma.py` as the current core extractor.
+
+## Extracted Indicators
+
+Supported Sigma fields:

 - `Image|endswith` -> `process_image_endswith`
 - `CommandLine|contains` -> `commandline_contains`
 - `OriginalFileName` -> `original_filename`
 - `Hashes|SHA256` -> `sha256`

-## Scripts
+The extractor de-duplicates and sorts rows before writing outputs.

- `scripts/extract_ioc_from_sigma.py` (core extractor)
- `scripts/build_dlp_ioc_from_hayabusa.sh` (wrapper)
+## Output Artifacts

-## Production (AW server <AW_SERVER_HOST>)
+Generated files:

-IOC enrichment is deployed by `ansible/deploy_aw_server.yml` when `aw_dlp_ioc_enabled=true`.
-
- systemd service: `aw-dlp-ioc-refresh.service`
- systemd timer: `aw-dlp-ioc-refresh.timer`
- refresh interval: `aw_dlp_ioc_refresh_interval` (default `6h`)
- output dir: `/opt/activitywatch/dlp-ioc/output`
- HTTP export via existing AW worktime API (`:5610`):
-  - `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.json`
-  - `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.csv`
-  - `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.sql`
-
-Mandatory post-deploy checks in Ansible:
 - `ioc_blacklist.json`
 - `ioc_blacklist.csv`
 - `ioc_blacklist.sql`

-Each file must exist and be non-empty, otherwise deploy fails.
+Production HTTP export through `aw-worktime-api` (`:5610`):

-## Run
+- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.json`
+- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.csv`
+- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.sql`

-```bash
-cd <PROJECT_ROOT>
-bash scripts/build_dlp_ioc_from_hayabusa.sh
+`aw-worktime-api` only serves these three IOC filenames from the DLP IOC
+directory.
+
+## Endpoint Consumption
+
+Windows DLP policy consumes the feed through the `ioc` block:
+
+```json
+{
+  "ioc": {
+    "enabled": true,
+    "source": "http://aw-server.example.local:5610/dlp-ioc/ioc_blacklist.json",
+    "format": "hayabusa_sigma_v1",
+    "refreshMinutes": 60
+  }
+}
 ```

-Optional custom paths:
+The endpoint collector loads this source and reports loaded IOC state in its
+health/heartbeat data, including `iocRulesLoaded`.
+
+## Operational Checks
+
+Server checks:

 ```bash
-bash scripts/build_dlp_ioc_from_hayabusa.sh \
-  /mnt/usb_hdd1/Projects/hayabusa/rules \
-  <PROJECT_ROOT>/data/dlp-ioc
+systemctl status aw-dlp-ioc-refresh.timer --no-pager
+systemctl status aw-dlp-ioc-refresh.service --no-pager
+journalctl -u aw-dlp-ioc-refresh.service -n 80 --no-pager
+ls -lh /opt/activitywatch/dlp-ioc/output/ioc_blacklist.*
+curl -fsS http://127.0.0.1:5610/dlp-ioc/ioc_blacklist.json | jq 'length'
 ```

-## Output artifacts
+Expected result:

- `data/dlp-ioc/ioc_blacklist.json`
- `data/dlp-ioc/ioc_blacklist.csv`
- `data/dlp-ioc/ioc_blacklist.sql`
+- timer is enabled and active;
+- last service run completed successfully;
+- `ioc_blacklist.json`, `ioc_blacklist.csv`, and `ioc_blacklist.sql` exist and
+  are non-empty;
+- Worktime API serves the JSON feed;
+- endpoint health shows non-zero `iocRulesLoaded` when the feed contains rules.

-## DLP import mapping
+Mandatory post-deploy checks in Ansible require all three output files to exist
+and be non-empty. Deployment fails if any artifact is missing or empty.

- `process_image_endswith` -> denied process/image list
- `commandline_contains` -> denied command pattern list
- `original_filename` -> suspicious original filename list
- `sha256` -> malware hash blocklist
+## Boundaries

-## Safety notes
+- This pipeline enriches DLP IOC/signature inputs automatically.
+- It does not approve, deploy, or roll back policy versions. That is the role
+  of the DLP Policy Engine.
+- It does not run Hayabusa against EVTX artifacts. That is the separate
+  server-side Hayabusa forensics path.
+- It does not modify running DLP agents directly; agents consume the published
+  IOC feed through their policy.

- This pipeline only creates export artifacts and does not modify running DLP agents.
- Review and tune false positives before enforcing blocking in production.
+Review and tune false positives before using generated indicators for blocking
+actions in production.
@@ -153,10 +153,16 @@ Deployment/tooling:

 Сценарии и артефакты:

- `scripts/extract_ioc_from_sigma.py`
-  Извлечение IOC из Sigma/Hayabusa rules.
+- `adk-rust/crates/extract-ioc-from-sigma`
+  Production Rust extractor для извлечения IOC из Sigma/Hayabusa rules.
+- `/usr/local/bin/aw-extract-ioc-from-sigma`
+  Установленный production binary на AW server.
 - `scripts/build_dlp_ioc_from_hayabusa.sh`
-  Построение JSON/CSV/SQL артефактов IOC.
+  Локальный/manual wrapper для построения JSON/CSV/SQL артефактов IOC.
+- `aw-dlp-ioc-refresh.service` / `aw-dlp-ioc-refresh.timer`
+  Автоматическое пополнение IOC из upstream `Yamato-Security/hayabusa-rules`.
+
+Подробный runtime contract: `docs/dlp-ioc-enrichment.md`.

 ### 2.9 Health / autoheal / operations

@@ -478,11 +484,18 @@ Policy Engine поддерживает:

 ## 15. IOC enrichment через Hayabusa / Sigma

-Реализован вспомогательный pipeline:
+Реализован автоматический pipeline пополнения DLP IOC/сигнатур:

- разбор Sigma/YAML правил;
+- источник: GitHub ruleset `Yamato-Security/hayabusa-rules`;
+- deployment variable: `aw_dlp_ioc_rules_zip_url`;
+- расписание: `aw-dlp-ioc-refresh.timer`, по умолчанию каждые `6h`;
+- production extractor: Rust binary `/usr/local/bin/aw-extract-ioc-from-sigma`;
+- разбор Sigma/YAML правил из `hayabusa-rules`;
 - извлечение IOC-полей;
- выгрузка в `json/csv/sql`.
+- выгрузка в `ioc_blacklist.json`, `ioc_blacklist.csv`, `ioc_blacklist.sql`;
+- публикация через `aw-worktime-api` на `/dlp-ioc/ioc_blacklist.*`;
+- потребление Windows DLP policy через `ioc.source` и формат
+  `hayabusa_sigma_v1`.

 Извлекаемые типы:

@@ -494,6 +507,15 @@ Policy Engine поддерживает:
 Назначение:

 - preload blacklist/indicator данных для DLP и смежной аналитики.
+- автоматическое обогащение DLP rules без ручного редактирования endpoint JSON.
+
+Границы:
+
+- DLP Policy Engine управляет жизненным циклом политик
+  (`draft/approve/deploy/rollback`), но не является источником upstream
+  сигнатур.
+- Hayabusa/Sigma IOC enrichment не равен Hayabusa EVTX forensics runner; это
+  отдельный контур пополнения IOC blacklist.

 ## 16. Health-check, autoheal и эксплуатационная устойчивость

@@ -84,6 +84,21 @@ IOC-слой позволяет подтягивать внешние индик
 - `format`
 - `refreshMinutes`

+В production-контуре DetMir этот слой заполняется автоматически через
+`DLP IOC Enrichment from Hayabusa/Sigma`:
+
+- upstream ruleset: `Yamato-Security/hayabusa-rules`;
+- Ansible URL: `aw_dlp_ioc_rules_zip_url`;
+- refresh: `aw-dlp-ioc-refresh.service` / `aw-dlp-ioc-refresh.timer`;
+- extractor: Rust binary `/usr/local/bin/aw-extract-ioc-from-sigma`;
+- published feed: `/dlp-ioc/ioc_blacklist.json`;
+- policy format: `hayabusa_sigma_v1`;
+- endpoint health field: `iocRulesLoaded`.
+
+Это не ручной ввод сигнатур в endpoint JSON. Endpoint policy только указывает
+`ioc.source`, а сами IOC blacklist artifacts генерируются на сервере из
+Hayabusa/Sigma rules.
+
 ## Действия

 На практике используются:
@@ -102,6 +117,7 @@ IOC-слой позволяет подтягивать внешние индик
 ## Канонические документы

 - [Пример policy](../../windows/dlp-policy.example.json)
+- [DLP IOC Enrichment from Hayabusa/Sigma](../dlp-ioc-enrichment.md)
 - [DLP Endpoint Monitoring](DLP-Endpoint-Monitoring)
 - [Email Outbound Monitoring](Email-Outbound-Monitoring)
 - [Категоризация сайтов](Web-Categorization)