From 61a9887ebbe90da6a63ca5e87fcd286995db8844 Mon Sep 17 00:00:00 2001 From: igor04091968 Date: Fri, 12 Jun 2026 03:46:43 +0300 Subject: [PATCH] docs: document dlp ioc enrichment pipeline --- AGENTS.md | 214 ++++++++++++++++++++++++ docs/dlp-ioc-enrichment.md | 170 ++++++++++++++----- docs/dlp-security-functional-spec-ru.md | 34 +++- docs/wiki/DLP-Rules.md | 16 ++ 4 files changed, 388 insertions(+), 46 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 85effe1..789d67c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,3 +30,217 @@ Operational rules for OpenCode/Codex agents in AWatch-rus. ## Entrypoints Use `proxmox/create-ct.sh`, `proxmox/push-aw-artifacts.sh`, `aw-server/install_aw_server.sh`, `aw-server/apply_webui_ru_patch.sh`, `windows/deploy-ensemble.ps1`, and docs in `docs/preparation.md`, `docs/deployment.md`, `docs/runbook.md`, `docs/operations.md`. + +## Incident Handling + +OpenCode must handle AWatch-rus incidents as evidence-based operational triage, +not as guesswork from one red dashboard card. + +### Assessment Basis + +Assess every incident from these signals, in this order: + +- **User impact:** portal/report/dashboard unavailable, stale, slow, or wrong; + which role is affected: executive, manager, security, forensics, admin. +- **Data freshness:** ActivityWatch bucket `metadata.end`, collector heartbeats, + Windows scheduled task recency, queue depth, and upload/send failure counters. +- **Service health:** systemd failed units, active timers, bounded HTTP checks, + `/health` or `/api/health` responses, container health where relevant. +- **Pipeline layer:** identify the first broken layer in the chain + `Windows/RDP collectors -> ActivityWatch buckets -> Rust services -> exporters + -> Grafana/Portal -> ClickHouse/1C where configured`. +- **Risk/evidence:** DLP endpoint signals, incident candidates, evidence + artifacts, UEBA/risk narrative inputs, coverage gaps, and security + correlation indicators. +- **Blast radius:** one user/session/collector, one host, one service, one + dashboard, or the full contour. +- **Recoverability:** known rollback, stale-cache availability, safe restart + boundary, and whether a human approval is required. + +Risk Narrative is only decision support. It can raise priority and explain +why a manual check is needed, but it does not prove a policy violation, DLP +incident, or SIEM finding by itself. + +### Severity + +Use this practical severity model: + +- `P0`: data loss risk, auth/security boundary broken, raw private service + exposed, production report chain unavailable with no stale fallback, or + repeated collector process storms/memory pressure. +- `P1`: executive/security workflows degraded, fresh data missing for a critical + host, DLP evidence sync broken, ClickHouse/1C ingest stopped, or portal health + degraded with user-visible effect. +- `P2`: one collector stale, one dashboard/panel wrong, delayed timer, bad label + normalization, missing noncritical evidence, or recoverable stale report. +- `P3`: documentation drift, cosmetic UI issue, non-production demo fixture, + or a warning with fresh data still confirmed. + +Escalate severity when the same symptom repeats after recovery, when coverage +is unknown, or when evidence contradicts dashboard status. + +### Mechanisms To Use + +Start with the repo wrappers before ad hoc probing: + +```bash +cd /mnt/usb_hdd2/Projects/ActivityWatch-Russian +./check-aw-data.sh +./check-aw-full.sh +``` + +Then narrow by layer: + +- ActivityWatch API: `/api/0/info`, `/api/0/buckets`, bucket metadata and recent + events with explicit `no_proxy` and short `curl --max-time`. +- Worktime: `aw-worktime-api` `/health`, `/reports/worktime/today`, + `/reports/worktime/management?allow_stale=1`, prewarm logs, stale-cache + fields, `AW_WORKTIME_EVENTS_LIMIT`, and `aw_query_timeout_count`. +- Windows/RDP: `validate-deployment.ps1`, exact `ActivityWatch Launch [...]` + scheduled tasks, `ActivityWatch Recovery`, collector guard state, session + collectors, local queue depth, and send failure counters. +- DLP: `aw-dlp-policy-engine`, `aw-dlp-case-management`, `dlp-health-check`, + `aw-dlp-endpoint-signals_`, evidence artifact sync, policy audit, and + case/compliance services. +- Portal/Gateway/Grafana: `/portal/api/health`, `/api/reports`, gateway + `/healthz`, protected `/d/...` Grafana routes, role gates, and browser smoke + scripts. +- ClickHouse/1C: only for file-1C/analytics incidents. Do not blame ClickHouse + for worktime report failures unless the affected path explicitly uses it. + +Use existing guards and bounded mechanisms before broad restarts: + +- stale-cache and fail-closed worktime behavior; +- `aw-worktime-autoheal`, `aw-worktime-prewarm`, `aw-worktime-ui-bridge`, + `aw-rus-healthd` timers; +- Windows collector guard and exact localized scheduled tasks; +- DLP evidence sync and health timers; +- targeted service restart only after evidence identifies the layer. + +### DLP Rule Update System + +Do not describe AWatch-rus DLP rules as manual local JSON entry, and do not +collapse all DLP updates into one mechanism. There are two related but separate +contours: + +1. policy lifecycle and endpoint synchronization through the DLP Policy Engine; +2. automatic IOC/signature replenishment from the open-source Hayabusa/Sigma + ruleset. + +The centralized policy update contour is: + +- Server service: `aw-dlp-policy-engine.service`, Rust binary + `/usr/local/bin/aw-dlp-policy-engine-rust`, default API port `5601`. +- Storage: SQLite DB from `AW_DLP_POLICY_ENGINE_DB_PATH`, with policy records, + policy versions, active policy pointer, rollback versions, and `policy_audit`. +- API contract: + - `GET /healthz`; + - CRUD: `/api/0/dlp/policies`; + - active bundle: `GET /api/0/dlp/policies/active`; + - active version/checksum: `GET /api/0/dlp/policies/active/version`; + - approval lifecycle: + `draft -> pending_approval -> approved -> deployed`; + - workflow calls: + `POST /submit`, `POST /approve`, `POST /draft`, `POST /activate`; + - rollback: `POST /api/0/dlp/policies/rollback`; + - audit: + `GET /api/0/dlp/policies/audit?limit=N` and + `GET /api/0/dlp/policies/{id}/audit?limit=N`; + - endpoint sync: + `POST /api/0/dlp/policies/agents/{agent_id}/heartbeat` and + `GET /api/0/dlp/policies/agents/{agent_id}/desired`. +- Windows side is configured for server-driven policy mode: + `aw_windows_policy_mode: "server"`, + `aw_windows_policy_engine_enabled: true`, + `aw_windows_policy_refresh_seconds: 300`, and policy engine host/port from + Ansible group vars. +- Agents report their current policy version/checksum by heartbeat. The server + compares it with the active deployed policy and returns `desired` with + `refreshNow=true` when the endpoint must update. +- `dlp-admin-cli` is the operator CLI for read-side checks such as + `policies list`, `policies active`, incident/case listing, and combined DLP + health checks. It is not a replacement for the lifecycle API when changing + policy state. + +Automatic IOC/signature replenishment: + +- Name it precisely as `DLP IOC Enrichment from Hayabusa/Sigma` or + `Hayabusa Sigma IOC refresh pipeline`. +- Source rules come from the open-source GitHub ruleset + `Yamato-Security/hayabusa-rules`, configured by + `aw_dlp_ioc_rules_zip_url`. +- Deployment is controlled by `ansible/deploy_aw_server.yml` when + `aw_dlp_ioc_enabled=true`. +- The refresh wrapper `/usr/local/bin/aw-dlp-ioc-refresh.sh` downloads the + latest `hayabusa-rules` ZIP, unpacks Sigma YAML rules, and runs the Rust + extractor `/usr/local/bin/aw-extract-ioc-from-sigma`. +- The Rust extractor is built from + `adk-rust/crates/extract-ioc-from-sigma`; local/manual builds use + `scripts/build_dlp_ioc_from_hayabusa.sh`. +- Extracted IOC-like values include process image suffixes, command-line + substrings, original filenames, and SHA256 hashes. They are de-duplicated and + emitted as `ioc_blacklist.json`, `ioc_blacklist.csv`, and + `ioc_blacklist.sql`. +- Production artifacts live under `/opt/activitywatch/dlp-ioc/output` and are + served by `aw-worktime-api` on `/dlp-ioc/ioc_blacklist.json`, + `/dlp-ioc/ioc_blacklist.csv`, and `/dlp-ioc/ioc_blacklist.sql`. +- Windows DLP policy can consume this feed through the `ioc.source` field with + format `hayabusa_sigma_v1`; endpoint health/heartbeat should expose loaded + IOC state such as `iocRulesLoaded`. +- Runtime automation is `aw-dlp-ioc-refresh.service` plus + `aw-dlp-ioc-refresh.timer` with interval `aw_dlp_ioc_refresh_interval` + (default `6h`). Health/diagnostics should check this timer before assuming + signatures are static or manually maintained. +- This Hayabusa/Sigma IOC pipeline enriches the DLP rule base automatically; it + is not the same thing as hand-editing endpoint JSON and is also distinct from + the server-side Hayabusa EVTX forensics runner. + +Operational meaning: + +1. To update rules, create or update a policy draft through the policy engine. +2. Submit it for approval, approve it, then activate/deploy it. Activation is + allowed only from `approved`. +3. For policy changes, verify `active/version`, audit entries, Windows agent + heartbeat/desired, and downstream DLP signals after endpoints refresh. +4. For automatic signature replenishment, verify + `aw-dlp-ioc-refresh.timer`, the last `aw-dlp-ioc-refresh.service` run, + non-empty `ioc_blacklist.json/csv/sql`, Worktime API `/dlp-ioc/...` + exports, and Windows IOC load counters. +5. If a policy causes noise or misses, use policy rollback through the API; do + not hand-edit endpoint policy files as the normal rollback path. + +Manual edits of `C:\Program Files\AWatch-rus\windows\dlp-policy.example.json` +or `C:\ProgramData\AWatch-rus\dlp-policy.json` are diagnostic or emergency +fallback only. If such an edit is unavoidable, document it as configuration +drift and bring the rule back into the central policy engine. + +### Response Workflow + +1. Capture current state first: command, timestamp, host, service, and exact + failing endpoint. Do not restart before collecting evidence unless the + system is in active resource exhaustion. +2. Find the first broken layer. If buckets are stale, fix collectors before + Grafana. If `aw-worktime-api` is degraded, fix/report that before portal. +3. Separate real outage from presentation drift: dashboards can be stale or + mislabeled while buckets and services are healthy. +4. Apply the narrowest safe recovery: restart a collector/task/service, reduce + unsafe limits, clear process storms, or restore a known-good binary/config. + Back up config/binaries before replacement. +5. Verify with the same failing check plus one upstream and one downstream + check. For collector incidents, require bucket freshness and guard/healthd + consistency, not just one green command. +6. Record closure evidence: root cause, affected layer, action taken, commands + run, post-check results, remaining risk, and rollback path. + +### Safety Rules + +- Old snapshots, memory, dashboards, and handoff notes are hints; live runtime + evidence wins. +- Never expose passwords, tokens, private host credentials, private URLs, raw + security events, or customer identifiers in incident writeups. +- Do not run broad deploys, full restarts, or `cargo build --workspace` during + incident triage unless the scope demands it and rollback is clear. +- Do not treat `status=ok` as sufficient when freshness, queue depth, or + coverage evidence says otherwise. +- For owner-facing reports, publish only protected gateway/Grafana routes, not + raw `:5600`, `:5610`, `:8720`, or ClickHouse endpoints. diff --git a/docs/dlp-ioc-enrichment.md b/docs/dlp-ioc-enrichment.md index d5353d9..c62771b 100644 --- a/docs/dlp-ioc-enrichment.md +++ b/docs/dlp-ioc-enrichment.md @@ -1,72 +1,162 @@ # DLP IOC Enrichment from Hayabusa/Sigma -This adds a safe offline pipeline to preload DLP blacklists from static Sigma indicators. +This document describes the automatic DLP IOC/signature replenishment pipeline. +It is separate from the DLP Policy Engine lifecycle and from the Hayabusa EVTX +forensics runner. + +## Purpose + +The pipeline preloads DLP indicator blacklists from static Sigma indicators. It +is used to enrich endpoint DLP rules without manually editing endpoint JSON +policy files. ## Source -- Sigma rules from Hayabusa ruleset (`hayabusa-rules` YAML files). +- Upstream ruleset: `Yamato-Security/hayabusa-rules` +- Default source URL: + `https://github.com/Yamato-Security/hayabusa-rules/archive/refs/heads/main.zip` +- Ansible variable: `aw_dlp_ioc_rules_zip_url` +- Production enable flag: `aw_dlp_ioc_enabled` -## Extracted indicators +Hayabusa rules may carry licenses that are separate from the Hayabusa binary. +Check the upstream ruleset license before packaging or redistributing generated +artifacts. + +## Production Pipeline + +When `aw_dlp_ioc_enabled=true`, `ansible/deploy_aw_server.yml` installs and +starts this chain on the AW server: + +1. `aw-dlp-ioc-refresh.timer` runs on boot and then every + `aw_dlp_ioc_refresh_interval` (`6h` by default). +2. The timer starts `aw-dlp-ioc-refresh.service`. +3. The service executes `/usr/local/bin/aw-dlp-ioc-refresh.sh`. +4. The wrapper downloads `aw_dlp_ioc_rules_zip_url`. +5. The wrapper unpacks `hayabusa-rules` Sigma YAML files into a temporary + working directory. +6. The wrapper runs `/usr/local/bin/aw-extract-ioc-from-sigma`. +7. The Rust extractor writes generated IOC artifacts to + `/opt/activitywatch/dlp-ioc/output`. +8. `aw-worktime-api` serves the artifacts from `/dlp-ioc/...` for DLP policy + consumption. + +Production units: + +- `aw-dlp-ioc-refresh.service` +- `aw-dlp-ioc-refresh.timer` + +Production paths: + +- workdir: `/opt/activitywatch/dlp-ioc` +- output dir: `/opt/activitywatch/dlp-ioc/output` +- latest symlink: `/opt/activitywatch/dlp-ioc/latest` + +## Extractor + +Primary extractor: + +- crate: `adk-rust/crates/extract-ioc-from-sigma` +- installed binary: `/usr/local/bin/aw-extract-ioc-from-sigma` +- local build: + +```bash +cd /adk-rust +cargo build --release -p extract-ioc-from-sigma +``` + +Local/manual wrapper: + +- `scripts/build_dlp_ioc_from_hayabusa.sh` + +```bash +cd +bash scripts/build_dlp_ioc_from_hayabusa.sh \ + /mnt/usb_hdd1/Projects/hayabusa/rules \ + /data/dlp-ioc +``` + +The old Python extractor path is not the production path. Do not document +`scripts/extract_ioc_from_sigma.py` as the current core extractor. + +## Extracted Indicators + +Supported Sigma fields: - `Image|endswith` -> `process_image_endswith` - `CommandLine|contains` -> `commandline_contains` - `OriginalFileName` -> `original_filename` - `Hashes|SHA256` -> `sha256` -## Scripts +The extractor de-duplicates and sorts rows before writing outputs. -- `scripts/extract_ioc_from_sigma.py` (core extractor) -- `scripts/build_dlp_ioc_from_hayabusa.sh` (wrapper) +## Output Artifacts -## Production (AW server ) +Generated files: -IOC enrichment is deployed by `ansible/deploy_aw_server.yml` when `aw_dlp_ioc_enabled=true`. - -- systemd service: `aw-dlp-ioc-refresh.service` -- systemd timer: `aw-dlp-ioc-refresh.timer` -- refresh interval: `aw_dlp_ioc_refresh_interval` (default `6h`) -- output dir: `/opt/activitywatch/dlp-ioc/output` -- HTTP export via existing AW worktime API (`:5610`): - - `http://:5610/dlp-ioc/ioc_blacklist.json` - - `http://:5610/dlp-ioc/ioc_blacklist.csv` - - `http://:5610/dlp-ioc/ioc_blacklist.sql` - -Mandatory post-deploy checks in Ansible: - `ioc_blacklist.json` - `ioc_blacklist.csv` - `ioc_blacklist.sql` -Each file must exist and be non-empty, otherwise deploy fails. +Production HTTP export through `aw-worktime-api` (`:5610`): -## Run +- `http://:5610/dlp-ioc/ioc_blacklist.json` +- `http://:5610/dlp-ioc/ioc_blacklist.csv` +- `http://:5610/dlp-ioc/ioc_blacklist.sql` -```bash -cd -bash scripts/build_dlp_ioc_from_hayabusa.sh +`aw-worktime-api` only serves these three IOC filenames from the DLP IOC +directory. + +## Endpoint Consumption + +Windows DLP policy consumes the feed through the `ioc` block: + +```json +{ + "ioc": { + "enabled": true, + "source": "http://aw-server.example.local:5610/dlp-ioc/ioc_blacklist.json", + "format": "hayabusa_sigma_v1", + "refreshMinutes": 60 + } +} ``` -Optional custom paths: +The endpoint collector loads this source and reports loaded IOC state in its +health/heartbeat data, including `iocRulesLoaded`. + +## Operational Checks + +Server checks: ```bash -bash scripts/build_dlp_ioc_from_hayabusa.sh \ - /mnt/usb_hdd1/Projects/hayabusa/rules \ - /data/dlp-ioc +systemctl status aw-dlp-ioc-refresh.timer --no-pager +systemctl status aw-dlp-ioc-refresh.service --no-pager +journalctl -u aw-dlp-ioc-refresh.service -n 80 --no-pager +ls -lh /opt/activitywatch/dlp-ioc/output/ioc_blacklist.* +curl -fsS http://127.0.0.1:5610/dlp-ioc/ioc_blacklist.json | jq 'length' ``` -## Output artifacts +Expected result: -- `data/dlp-ioc/ioc_blacklist.json` -- `data/dlp-ioc/ioc_blacklist.csv` -- `data/dlp-ioc/ioc_blacklist.sql` +- timer is enabled and active; +- last service run completed successfully; +- `ioc_blacklist.json`, `ioc_blacklist.csv`, and `ioc_blacklist.sql` exist and + are non-empty; +- Worktime API serves the JSON feed; +- endpoint health shows non-zero `iocRulesLoaded` when the feed contains rules. -## DLP import mapping +Mandatory post-deploy checks in Ansible require all three output files to exist +and be non-empty. Deployment fails if any artifact is missing or empty. -- `process_image_endswith` -> denied process/image list -- `commandline_contains` -> denied command pattern list -- `original_filename` -> suspicious original filename list -- `sha256` -> malware hash blocklist +## Boundaries -## Safety notes +- This pipeline enriches DLP IOC/signature inputs automatically. +- It does not approve, deploy, or roll back policy versions. That is the role + of the DLP Policy Engine. +- It does not run Hayabusa against EVTX artifacts. That is the separate + server-side Hayabusa forensics path. +- It does not modify running DLP agents directly; agents consume the published + IOC feed through their policy. -- This pipeline only creates export artifacts and does not modify running DLP agents. -- Review and tune false positives before enforcing blocking in production. +Review and tune false positives before using generated indicators for blocking +actions in production. diff --git a/docs/dlp-security-functional-spec-ru.md b/docs/dlp-security-functional-spec-ru.md index 1cf521d..90e5f6c 100644 --- a/docs/dlp-security-functional-spec-ru.md +++ b/docs/dlp-security-functional-spec-ru.md @@ -153,10 +153,16 @@ Deployment/tooling: Сценарии и артефакты: -- `scripts/extract_ioc_from_sigma.py` - Извлечение IOC из Sigma/Hayabusa rules. +- `adk-rust/crates/extract-ioc-from-sigma` + Production Rust extractor для извлечения IOC из Sigma/Hayabusa rules. +- `/usr/local/bin/aw-extract-ioc-from-sigma` + Установленный production binary на AW server. - `scripts/build_dlp_ioc_from_hayabusa.sh` - Построение JSON/CSV/SQL артефактов IOC. + Локальный/manual wrapper для построения JSON/CSV/SQL артефактов IOC. +- `aw-dlp-ioc-refresh.service` / `aw-dlp-ioc-refresh.timer` + Автоматическое пополнение IOC из upstream `Yamato-Security/hayabusa-rules`. + +Подробный runtime contract: `docs/dlp-ioc-enrichment.md`. ### 2.9 Health / autoheal / operations @@ -478,11 +484,18 @@ Policy Engine поддерживает: ## 15. IOC enrichment через Hayabusa / Sigma -Реализован вспомогательный pipeline: +Реализован автоматический pipeline пополнения DLP IOC/сигнатур: -- разбор Sigma/YAML правил; +- источник: GitHub ruleset `Yamato-Security/hayabusa-rules`; +- deployment variable: `aw_dlp_ioc_rules_zip_url`; +- расписание: `aw-dlp-ioc-refresh.timer`, по умолчанию каждые `6h`; +- production extractor: Rust binary `/usr/local/bin/aw-extract-ioc-from-sigma`; +- разбор Sigma/YAML правил из `hayabusa-rules`; - извлечение IOC-полей; -- выгрузка в `json/csv/sql`. +- выгрузка в `ioc_blacklist.json`, `ioc_blacklist.csv`, `ioc_blacklist.sql`; +- публикация через `aw-worktime-api` на `/dlp-ioc/ioc_blacklist.*`; +- потребление Windows DLP policy через `ioc.source` и формат + `hayabusa_sigma_v1`. Извлекаемые типы: @@ -494,6 +507,15 @@ Policy Engine поддерживает: Назначение: - preload blacklist/indicator данных для DLP и смежной аналитики. +- автоматическое обогащение DLP rules без ручного редактирования endpoint JSON. + +Границы: + +- DLP Policy Engine управляет жизненным циклом политик + (`draft/approve/deploy/rollback`), но не является источником upstream + сигнатур. +- Hayabusa/Sigma IOC enrichment не равен Hayabusa EVTX forensics runner; это + отдельный контур пополнения IOC blacklist. ## 16. Health-check, autoheal и эксплуатационная устойчивость diff --git a/docs/wiki/DLP-Rules.md b/docs/wiki/DLP-Rules.md index a3d43db..68c46f5 100644 --- a/docs/wiki/DLP-Rules.md +++ b/docs/wiki/DLP-Rules.md @@ -84,6 +84,21 @@ IOC-слой позволяет подтягивать внешние индик - `format` - `refreshMinutes` +В production-контуре DetMir этот слой заполняется автоматически через +`DLP IOC Enrichment from Hayabusa/Sigma`: + +- upstream ruleset: `Yamato-Security/hayabusa-rules`; +- Ansible URL: `aw_dlp_ioc_rules_zip_url`; +- refresh: `aw-dlp-ioc-refresh.service` / `aw-dlp-ioc-refresh.timer`; +- extractor: Rust binary `/usr/local/bin/aw-extract-ioc-from-sigma`; +- published feed: `/dlp-ioc/ioc_blacklist.json`; +- policy format: `hayabusa_sigma_v1`; +- endpoint health field: `iocRulesLoaded`. + +Это не ручной ввод сигнатур в endpoint JSON. Endpoint policy только указывает +`ioc.source`, а сами IOC blacklist artifacts генерируются на сервере из +Hayabusa/Sigma rules. + ## Действия На практике используются: @@ -102,6 +117,7 @@ IOC-слой позволяет подтягивать внешние индик ## Канонические документы - [Пример policy](../../windows/dlp-policy.example.json) +- [DLP IOC Enrichment from Hayabusa/Sigma](../dlp-ioc-enrichment.md) - [DLP Endpoint Monitoring](DLP-Endpoint-Monitoring) - [Email Outbound Monitoring](Email-Outbound-Monitoring) - [Категоризация сайтов](Web-Categorization)