awatch-rus/AWatch-rus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document dlp ioc enrichment pipeline

Browse Source
This commit is contained in:
igor04091968
2026-06-12 03:46:43 +03:00
parent aece5c381f
commit 61a9887ebb
4 changed files with 388 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+214
View File
@@ -30,3 +30,217 @@ Operational rules for OpenCode/Codex agents in AWatch-rus.
## Entrypoints
Use `proxmox/create-ct.sh`, `proxmox/push-aw-artifacts.sh`, `aw-server/install_aw_server.sh`, `aw-server/apply_webui_ru_patch.sh`, `windows/deploy-ensemble.ps1`, and docs in `docs/preparation.md`, `docs/deployment.md`, `docs/runbook.md`, `docs/operations.md`.
## Incident Handling
OpenCode must handle AWatch-rus incidents as evidence-based operational triage,
not as guesswork from one red dashboard card.
### Assessment Basis
Assess every incident from these signals, in this order:
- **User impact:** portal/report/dashboard unavailable, stale, slow, or wrong;
which role is affected: executive, manager, security, forensics, admin.
- **Data freshness:** ActivityWatch bucket `metadata.end`, collector heartbeats,
Windows scheduled task recency, queue depth, and upload/send failure counters.
- **Service health:** systemd failed units, active timers, bounded HTTP checks,
`/health` or `/api/health` responses, container health where relevant.
- **Pipeline layer:** identify the first broken layer in the chain
`Windows/RDP collectors -> ActivityWatch buckets -> Rust services -> exporters
-> Grafana/Portal -> ClickHouse/1C where configured`.
- **Risk/evidence:** DLP endpoint signals, incident candidates, evidence
artifacts, UEBA/risk narrative inputs, coverage gaps, and security
correlation indicators.
- **Blast radius:** one user/session/collector, one host, one service, one
dashboard, or the full contour.
- **Recoverability:** known rollback, stale-cache availability, safe restart
boundary, and whether a human approval is required.
Risk Narrative is only decision support. It can raise priority and explain
why a manual check is needed, but it does not prove a policy violation, DLP
incident, or SIEM finding by itself.
### Severity
Use this practical severity model:
- `P0`: data loss risk, auth/security boundary broken, raw private service
exposed, production report chain unavailable with no stale fallback, or
repeated collector process storms/memory pressure.
- `P1`: executive/security workflows degraded, fresh data missing for a critical
host, DLP evidence sync broken, ClickHouse/1C ingest stopped, or portal health
degraded with user-visible effect.
- `P2`: one collector stale, one dashboard/panel wrong, delayed timer, bad label
normalization, missing noncritical evidence, or recoverable stale report.
- `P3`: documentation drift, cosmetic UI issue, non-production demo fixture,
or a warning with fresh data still confirmed.
Escalate severity when the same symptom repeats after recovery, when coverage
is unknown, or when evidence contradicts dashboard status.
### Mechanisms To Use
Start with the repo wrappers before ad hoc probing:
```bash
cd /mnt/usb_hdd2/Projects/ActivityWatch-Russian
./check-aw-data.sh
./check-aw-full.sh
```
Then narrow by layer:
- ActivityWatch API: `/api/0/info`, `/api/0/buckets`, bucket metadata and recent
events with explicit `no_proxy` and short `curl --max-time`.
- Worktime: `aw-worktime-api` `/health`, `/reports/worktime/today`,
`/reports/worktime/management?allow_stale=1`, prewarm logs, stale-cache
fields, `AW_WORKTIME_EVENTS_LIMIT`, and `aw_query_timeout_count`.
- Windows/RDP: `validate-deployment.ps1`, exact `ActivityWatch Launch [...]`
scheduled tasks, `ActivityWatch Recovery`, collector guard state, session
collectors, local queue depth, and send failure counters.
- DLP: `aw-dlp-policy-engine`, `aw-dlp-case-management`, `dlp-health-check`,
`aw-dlp-endpoint-signals_<HOST>`, evidence artifact sync, policy audit, and
case/compliance services.
- Portal/Gateway/Grafana: `/portal/api/health`, `/api/reports`, gateway
`/healthz`, protected `/d/...` Grafana routes, role gates, and browser smoke
scripts.
- ClickHouse/1C: only for file-1C/analytics incidents. Do not blame ClickHouse
for worktime report failures unless the affected path explicitly uses it.
Use existing guards and bounded mechanisms before broad restarts:
- stale-cache and fail-closed worktime behavior;
- `aw-worktime-autoheal`, `aw-worktime-prewarm`, `aw-worktime-ui-bridge`,
`aw-rus-healthd` timers;
- Windows collector guard and exact localized scheduled tasks;
- DLP evidence sync and health timers;
- targeted service restart only after evidence identifies the layer.
### DLP Rule Update System
Do not describe AWatch-rus DLP rules as manual local JSON entry, and do not
collapse all DLP updates into one mechanism. There are two related but separate
contours:
1. policy lifecycle and endpoint synchronization through the DLP Policy Engine;
2. automatic IOC/signature replenishment from the open-source Hayabusa/Sigma
ruleset.
The centralized policy update contour is:
- Server service: `aw-dlp-policy-engine.service`, Rust binary
`/usr/local/bin/aw-dlp-policy-engine-rust`, default API port `5601`.
- Storage: SQLite DB from `AW_DLP_POLICY_ENGINE_DB_PATH`, with policy records,
policy versions, active policy pointer, rollback versions, and `policy_audit`.
- API contract:
- `GET /healthz`;
- CRUD: `/api/0/dlp/policies`;
- active bundle: `GET /api/0/dlp/policies/active`;
- active version/checksum: `GET /api/0/dlp/policies/active/version`;
- approval lifecycle:
`draft -> pending_approval -> approved -> deployed`;
- workflow calls:
`POST /submit`, `POST /approve`, `POST /draft`, `POST /activate`;
- rollback: `POST /api/0/dlp/policies/rollback`;
- audit:
`GET /api/0/dlp/policies/audit?limit=N` and
`GET /api/0/dlp/policies/{id}/audit?limit=N`;
- endpoint sync:
`POST /api/0/dlp/policies/agents/{agent_id}/heartbeat` and
`GET /api/0/dlp/policies/agents/{agent_id}/desired`.
- Windows side is configured for server-driven policy mode:
`aw_windows_policy_mode: "server"`,
`aw_windows_policy_engine_enabled: true`,
`aw_windows_policy_refresh_seconds: 300`, and policy engine host/port from
Ansible group vars.
- Agents report their current policy version/checksum by heartbeat. The server
compares it with the active deployed policy and returns `desired` with
`refreshNow=true` when the endpoint must update.
- `dlp-admin-cli` is the operator CLI for read-side checks such as
`policies list`, `policies active`, incident/case listing, and combined DLP
health checks. It is not a replacement for the lifecycle API when changing
policy state.
Automatic IOC/signature replenishment:
- Name it precisely as `DLP IOC Enrichment from Hayabusa/Sigma` or
`Hayabusa Sigma IOC refresh pipeline`.
- Source rules come from the open-source GitHub ruleset
`Yamato-Security/hayabusa-rules`, configured by
`aw_dlp_ioc_rules_zip_url`.
- Deployment is controlled by `ansible/deploy_aw_server.yml` when
`aw_dlp_ioc_enabled=true`.
- The refresh wrapper `/usr/local/bin/aw-dlp-ioc-refresh.sh` downloads the
latest `hayabusa-rules` ZIP, unpacks Sigma YAML rules, and runs the Rust
extractor `/usr/local/bin/aw-extract-ioc-from-sigma`.
- The Rust extractor is built from
`adk-rust/crates/extract-ioc-from-sigma`; local/manual builds use
`scripts/build_dlp_ioc_from_hayabusa.sh`.
- Extracted IOC-like values include process image suffixes, command-line
substrings, original filenames, and SHA256 hashes. They are de-duplicated and
emitted as `ioc_blacklist.json`, `ioc_blacklist.csv`, and
`ioc_blacklist.sql`.
- Production artifacts live under `/opt/activitywatch/dlp-ioc/output` and are
served by `aw-worktime-api` on `/dlp-ioc/ioc_blacklist.json`,
`/dlp-ioc/ioc_blacklist.csv`, and `/dlp-ioc/ioc_blacklist.sql`.
- Windows DLP policy can consume this feed through the `ioc.source` field with
format `hayabusa_sigma_v1`; endpoint health/heartbeat should expose loaded
IOC state such as `iocRulesLoaded`.
- Runtime automation is `aw-dlp-ioc-refresh.service` plus
`aw-dlp-ioc-refresh.timer` with interval `aw_dlp_ioc_refresh_interval`
(default `6h`). Health/diagnostics should check this timer before assuming
signatures are static or manually maintained.
- This Hayabusa/Sigma IOC pipeline enriches the DLP rule base automatically; it
is not the same thing as hand-editing endpoint JSON and is also distinct from
the server-side Hayabusa EVTX forensics runner.
Operational meaning:
1. To update rules, create or update a policy draft through the policy engine.
2. Submit it for approval, approve it, then activate/deploy it. Activation is
allowed only from `approved`.
3. For policy changes, verify `active/version`, audit entries, Windows agent
heartbeat/desired, and downstream DLP signals after endpoints refresh.
4. For automatic signature replenishment, verify
`aw-dlp-ioc-refresh.timer`, the last `aw-dlp-ioc-refresh.service` run,
non-empty `ioc_blacklist.json/csv/sql`, Worktime API `/dlp-ioc/...`
exports, and Windows IOC load counters.
5. If a policy causes noise or misses, use policy rollback through the API; do
not hand-edit endpoint policy files as the normal rollback path.
Manual edits of `C:\Program Files\AWatch-rus\windows\dlp-policy.example.json`
or `C:\ProgramData\AWatch-rus\dlp-policy.json` are diagnostic or emergency
fallback only. If such an edit is unavoidable, document it as configuration
drift and bring the rule back into the central policy engine.
### Response Workflow
1. Capture current state first: command, timestamp, host, service, and exact
failing endpoint. Do not restart before collecting evidence unless the
system is in active resource exhaustion.
2. Find the first broken layer. If buckets are stale, fix collectors before
Grafana. If `aw-worktime-api` is degraded, fix/report that before portal.
3. Separate real outage from presentation drift: dashboards can be stale or
mislabeled while buckets and services are healthy.
4. Apply the narrowest safe recovery: restart a collector/task/service, reduce
unsafe limits, clear process storms, or restore a known-good binary/config.
Back up config/binaries before replacement.
5. Verify with the same failing check plus one upstream and one downstream
check. For collector incidents, require bucket freshness and guard/healthd
consistency, not just one green command.
6. Record closure evidence: root cause, affected layer, action taken, commands
run, post-check results, remaining risk, and rollback path.
### Safety Rules
- Old snapshots, memory, dashboards, and handoff notes are hints; live runtime
evidence wins.
- Never expose passwords, tokens, private host credentials, private URLs, raw
security events, or customer identifiers in incident writeups.
- Do not run broad deploys, full restarts, or `cargo build --workspace` during
incident triage unless the scope demands it and rollback is clear.
- Do not treat `status=ok` as sufficient when freshness, queue depth, or
coverage evidence says otherwise.
- For owner-facing reports, publish only protected gateway/Grafana routes, not
raw `:5600`, `:5610`, `:8720`, or ClickHouse endpoints.
docs/dlp-ioc-enrichment.md
+130 -40
View File
@@ -1,72 +1,162 @@
# DLP IOC Enrichment from Hayabusa/Sigma
This adds a safe offline pipeline to preload DLP blacklists from static Sigma indicators.
This document describes the automatic DLP IOC/signature replenishment pipeline.
It is separate from the DLP Policy Engine lifecycle and from the Hayabusa EVTX
forensics runner.
## Purpose
The pipeline preloads DLP indicator blacklists from static Sigma indicators. It
is used to enrich endpoint DLP rules without manually editing endpoint JSON
policy files.
## Source
- Sigma rules from Hayabusa ruleset (`hayabusa-rules` YAML files).
- Upstream ruleset: `Yamato-Security/hayabusa-rules`
- Default source URL:
`https://github.com/Yamato-Security/hayabusa-rules/archive/refs/heads/main.zip`
- Ansible variable: `aw_dlp_ioc_rules_zip_url`
- Production enable flag: `aw_dlp_ioc_enabled`
## Extracted indicators
Hayabusa rules may carry licenses that are separate from the Hayabusa binary.
Check the upstream ruleset license before packaging or redistributing generated
artifacts.
## Production Pipeline
When `aw_dlp_ioc_enabled=true`, `ansible/deploy_aw_server.yml` installs and
starts this chain on the AW server:
1. `aw-dlp-ioc-refresh.timer` runs on boot and then every
`aw_dlp_ioc_refresh_interval` (`6h` by default).
2. The timer starts `aw-dlp-ioc-refresh.service`.
3. The service executes `/usr/local/bin/aw-dlp-ioc-refresh.sh`.
4. The wrapper downloads `aw_dlp_ioc_rules_zip_url`.
5. The wrapper unpacks `hayabusa-rules` Sigma YAML files into a temporary
working directory.
6. The wrapper runs `/usr/local/bin/aw-extract-ioc-from-sigma`.
7. The Rust extractor writes generated IOC artifacts to
`/opt/activitywatch/dlp-ioc/output`.
8. `aw-worktime-api` serves the artifacts from `/dlp-ioc/...` for DLP policy
consumption.
Production units:
- `aw-dlp-ioc-refresh.service`
- `aw-dlp-ioc-refresh.timer`
Production paths:
- workdir: `/opt/activitywatch/dlp-ioc`
- output dir: `/opt/activitywatch/dlp-ioc/output`
- latest symlink: `/opt/activitywatch/dlp-ioc/latest`
## Extractor
Primary extractor:
- crate: `adk-rust/crates/extract-ioc-from-sigma`
- installed binary: `/usr/local/bin/aw-extract-ioc-from-sigma`
- local build:
```bash
cd <PROJECT_ROOT>/adk-rust
cargo build --release -p extract-ioc-from-sigma
```
Local/manual wrapper:
- `scripts/build_dlp_ioc_from_hayabusa.sh`
```bash
cd <PROJECT_ROOT>
bash scripts/build_dlp_ioc_from_hayabusa.sh \
/mnt/usb_hdd1/Projects/hayabusa/rules \
<PROJECT_ROOT>/data/dlp-ioc
```
The old Python extractor path is not the production path. Do not document
`scripts/extract_ioc_from_sigma.py` as the current core extractor.
## Extracted Indicators
Supported Sigma fields:
- `Image|endswith` -> `process_image_endswith`
- `CommandLine|contains` -> `commandline_contains`
- `OriginalFileName` -> `original_filename`
- `Hashes|SHA256` -> `sha256`
## Scripts
The extractor de-duplicates and sorts rows before writing outputs.
- `scripts/extract_ioc_from_sigma.py` (core extractor)
- `scripts/build_dlp_ioc_from_hayabusa.sh` (wrapper)
## Output Artifacts
## Production (AW server <AW_SERVER_HOST>)
Generated files:
IOC enrichment is deployed by `ansible/deploy_aw_server.yml` when `aw_dlp_ioc_enabled=true`.
- systemd service: `aw-dlp-ioc-refresh.service`
- systemd timer: `aw-dlp-ioc-refresh.timer`
- refresh interval: `aw_dlp_ioc_refresh_interval` (default `6h`)
- output dir: `/opt/activitywatch/dlp-ioc/output`
- HTTP export via existing AW worktime API (`:5610`):
- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.json`
- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.csv`
- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.sql`
Mandatory post-deploy checks in Ansible:
- `ioc_blacklist.json`
- `ioc_blacklist.csv`
- `ioc_blacklist.sql`
Each file must exist and be non-empty, otherwise deploy fails.
Production HTTP export through `aw-worktime-api` (`:5610`):
## Run
- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.json`
- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.csv`
- `http://<AW_SERVER_HOST>:5610/dlp-ioc/ioc_blacklist.sql`
```bash
cd <PROJECT_ROOT>
bash scripts/build_dlp_ioc_from_hayabusa.sh
`aw-worktime-api` only serves these three IOC filenames from the DLP IOC
directory.
## Endpoint Consumption
Windows DLP policy consumes the feed through the `ioc` block:
```json
{
"ioc": {
"enabled": true,
"source": "http://aw-server.example.local:5610/dlp-ioc/ioc_blacklist.json",
"format": "hayabusa_sigma_v1",
"refreshMinutes": 60
}
}
```
Optional custom paths:
The endpoint collector loads this source and reports loaded IOC state in its
health/heartbeat data, including `iocRulesLoaded`.
## Operational Checks
Server checks:
```bash
bash scripts/build_dlp_ioc_from_hayabusa.sh \
/mnt/usb_hdd1/Projects/hayabusa/rules \
<PROJECT_ROOT>/data/dlp-ioc
systemctl status aw-dlp-ioc-refresh.timer --no-pager
systemctl status aw-dlp-ioc-refresh.service --no-pager
journalctl -u aw-dlp-ioc-refresh.service -n 80 --no-pager
ls -lh /opt/activitywatch/dlp-ioc/output/ioc_blacklist.*
curl -fsS http://127.0.0.1:5610/dlp-ioc/ioc_blacklist.json | jq 'length'
```
## Output artifacts
Expected result:
- `data/dlp-ioc/ioc_blacklist.json`
- `data/dlp-ioc/ioc_blacklist.csv`
- `data/dlp-ioc/ioc_blacklist.sql`
- timer is enabled and active;
- last service run completed successfully;
- `ioc_blacklist.json`, `ioc_blacklist.csv`, and `ioc_blacklist.sql` exist and
are non-empty;
- Worktime API serves the JSON feed;
- endpoint health shows non-zero `iocRulesLoaded` when the feed contains rules.
## DLP import mapping
Mandatory post-deploy checks in Ansible require all three output files to exist
and be non-empty. Deployment fails if any artifact is missing or empty.
- `process_image_endswith` -> denied process/image list
- `commandline_contains` -> denied command pattern list
- `original_filename` -> suspicious original filename list
- `sha256` -> malware hash blocklist
## Boundaries
## Safety notes
- This pipeline enriches DLP IOC/signature inputs automatically.
- It does not approve, deploy, or roll back policy versions. That is the role
of the DLP Policy Engine.
- It does not run Hayabusa against EVTX artifacts. That is the separate
server-side Hayabusa forensics path.
- It does not modify running DLP agents directly; agents consume the published
IOC feed through their policy.
- This pipeline only creates export artifacts and does not modify running DLP agents.
- Review and tune false positives before enforcing blocking in production.
Review and tune false positives before using generated indicators for blocking
actions in production.
docs/dlp-security-functional-spec-ru.md
+28 -6
View File
@@ -153,10 +153,16 @@ Deployment/tooling:
Сценарии и артефакты:
- `scripts/extract_ioc_from_sigma.py`
Извлечение IOC из Sigma/Hayabusa rules.
- `adk-rust/crates/extract-ioc-from-sigma`
Production Rust extractor для извлечения IOC из Sigma/Hayabusa rules.
- `/usr/local/bin/aw-extract-ioc-from-sigma`
Установленный production binary на AW server.
- `scripts/build_dlp_ioc_from_hayabusa.sh`
Построение JSON/CSV/SQL артефактов IOC.
Локальный/manual wrapper для построения JSON/CSV/SQL артефактов IOC.
- `aw-dlp-ioc-refresh.service` / `aw-dlp-ioc-refresh.timer`
Автоматическое пополнение IOC из upstream `Yamato-Security/hayabusa-rules`.
Подробный runtime contract: `docs/dlp-ioc-enrichment.md`.
### 2.9 Health / autoheal / operations
@@ -478,11 +484,18 @@ Policy Engine поддерживает:
## 15. IOC enrichment через Hayabusa / Sigma
Реализован вспомогательный pipeline:
Реализован автоматический pipeline пополнения DLP IOC/сигнатур:
- разбор Sigma/YAML правил;
- источник: GitHub ruleset `Yamato-Security/hayabusa-rules`;
- deployment variable: `aw_dlp_ioc_rules_zip_url`;
- расписание: `aw-dlp-ioc-refresh.timer`, по умолчанию каждые `6h`;
- production extractor: Rust binary `/usr/local/bin/aw-extract-ioc-from-sigma`;
- разбор Sigma/YAML правил из `hayabusa-rules`;
- извлечение IOC-полей;
- выгрузка в `json/csv/sql`.
- выгрузка в `ioc_blacklist.json`, `ioc_blacklist.csv`, `ioc_blacklist.sql`;
- публикация через `aw-worktime-api` на `/dlp-ioc/ioc_blacklist.*`;
- потребление Windows DLP policy через `ioc.source` и формат
`hayabusa_sigma_v1`.
Извлекаемые типы:
@@ -494,6 +507,15 @@ Policy Engine поддерживает:
Назначение:
- preload blacklist/indicator данных для DLP и смежной аналитики.
- автоматическое обогащение DLP rules без ручного редактирования endpoint JSON.
Границы:
- DLP Policy Engine управляет жизненным циклом политик
(`draft/approve/deploy/rollback`), но не является источником upstream
сигнатур.
- Hayabusa/Sigma IOC enrichment не равен Hayabusa EVTX forensics runner; это
отдельный контур пополнения IOC blacklist.
## 16. Health-check, autoheal и эксплуатационная устойчивость
docs/wiki/DLP-Rules.md
+16
View File
@@ -84,6 +84,21 @@ IOC-слой позволяет подтягивать внешние индик
- `format`
- `refreshMinutes`
В production-контуре DetMir этот слой заполняется автоматически через
`DLP IOC Enrichment from Hayabusa/Sigma`:
- upstream ruleset: `Yamato-Security/hayabusa-rules`;
- Ansible URL: `aw_dlp_ioc_rules_zip_url`;
- refresh: `aw-dlp-ioc-refresh.service` / `aw-dlp-ioc-refresh.timer`;
- extractor: Rust binary `/usr/local/bin/aw-extract-ioc-from-sigma`;
- published feed: `/dlp-ioc/ioc_blacklist.json`;
- policy format: `hayabusa_sigma_v1`;
- endpoint health field: `iocRulesLoaded`.
Это не ручной ввод сигнатур в endpoint JSON. Endpoint policy только указывает
`ioc.source`, а сами IOC blacklist artifacts генерируются на сервере из
Hayabusa/Sigma rules.
## Действия
На практике используются:
@@ -102,6 +117,7 @@ IOC-слой позволяет подтягивать внешние индик
## Канонические документы
- [Пример policy](../../windows/dlp-policy.example.json)
- [DLP IOC Enrichment from Hayabusa/Sigma](../dlp-ioc-enrichment.md)
- [DLP Endpoint Monitoring](DLP-Endpoint-Monitoring)
- [Email Outbound Monitoring](Email-Outbound-Monitoring)
- [Категоризация сайтов](Web-Categorization)