Add phased DLP reliability roadmap and acceptance criteria
This commit is contained in:
IgorRachkov
@@ -0,0 +1,151 @@
|
||||
# DLP Reliability Roadmap
|
||||
|
||||
## Scope
|
||||
Roadmap for improving runtime reliability of:
|
||||
- `windows/dlp-endpoint-signals-collector.ps1`
|
||||
- `windows/file-operations-collector.ps1`
|
||||
|
||||
Date: 2026-05-04
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 (1-2 days): Quick wins
|
||||
|
||||
### 1) Disk queue + sender loop + retry/backoff/jitter
|
||||
|
||||
**Goal:** no data loss on temporary network/server outages.
|
||||
|
||||
**Tasks**
|
||||
- Add local append-only queue file per collector (`*.jsonl`) under ProgramData logs/artifacts root.
|
||||
- Write events to queue first, then send asynchronously.
|
||||
- Implement sender loop:
|
||||
- reads oldest unsent records,
|
||||
- sends in small batches,
|
||||
- marks sent records,
|
||||
- compacts queue periodically.
|
||||
- Implement retry policy with exponential backoff + jitter.
|
||||
|
||||
**Acceptance criteria**
|
||||
- When API is unavailable, queue grows and collector keeps running.
|
||||
- When API recovers, queued events are flushed automatically.
|
||||
- No collector crash during repeated network failures.
|
||||
|
||||
### 2) `eventId` + dedupe contract
|
||||
|
||||
**Goal:** at-least-once delivery without logical duplicates.
|
||||
|
||||
**Tasks**
|
||||
- Add `eventId` (UUID), `eventCreatedAt`, `collectorType`, `hostname` to every payload.
|
||||
- Define server dedupe contract:
|
||||
- dedupe key = `eventId`,
|
||||
- TTL for dedupe cache,
|
||||
- idempotent processing semantics.
|
||||
|
||||
**Acceptance criteria**
|
||||
- Retried sends do not create duplicate incidents/events in downstream storage.
|
||||
- Payload schema documentation updated.
|
||||
|
||||
### 3) Basic metrics/logging
|
||||
|
||||
**Goal:** visibility into health and data delivery.
|
||||
|
||||
**Tasks**
|
||||
- Emit counters/gauges to log and heartbeat:
|
||||
- `queueDepth`,
|
||||
- `oldestUnsentAgeSec`,
|
||||
- `eventsEnqueued`,
|
||||
- `eventsSent`,
|
||||
- `sendFailures`,
|
||||
- `lastSendStatus`.
|
||||
|
||||
**Acceptance criteria**
|
||||
- Operators can identify stuck queue and send failures from logs only.
|
||||
|
||||
---
|
||||
|
||||
## Stage 2: Hardening
|
||||
|
||||
### 1) Circuit breaker + health probes
|
||||
|
||||
**Tasks**
|
||||
- Add transport circuit breaker (Closed/Open/HalfOpen).
|
||||
- Open breaker after N consecutive failures.
|
||||
- In Open state perform probe every M seconds.
|
||||
- Close breaker on successful probe.
|
||||
|
||||
**Acceptance criteria**
|
||||
- Reduced request storm during outage.
|
||||
- Deterministic recovery behavior after outage.
|
||||
|
||||
### 2) Watcher auto-recreate
|
||||
|
||||
**Tasks**
|
||||
- Handle `FileSystemWatcher` error/overflow events.
|
||||
- Recreate watcher and subscriptions automatically.
|
||||
- Keep watchdog timer to ensure watcher health.
|
||||
|
||||
**Acceptance criteria**
|
||||
- Watcher resumes after overflow without manual restart.
|
||||
|
||||
### 3) Last-known-good policy
|
||||
|
||||
**Tasks**
|
||||
- Validate new policy before apply.
|
||||
- Cache last valid policy with checksum/version.
|
||||
- Rollback to cached policy on parse/validation errors.
|
||||
|
||||
**Acceptance criteria**
|
||||
- Broken policy cannot stop detection loop.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3: Reliability operations
|
||||
|
||||
### 1) Chaos tests
|
||||
|
||||
Scenarios:
|
||||
- network disconnect,
|
||||
- API 5xx bursts,
|
||||
- slow disk / queue write delay,
|
||||
- headless UI context,
|
||||
- forced collector restart.
|
||||
|
||||
**Acceptance criteria**
|
||||
- For each scenario, documented expected behavior and observed result.
|
||||
- No silent data loss in tested outage windows.
|
||||
|
||||
### 2) SLO + error budget process
|
||||
|
||||
**Initial SLO proposals**
|
||||
- Event delivery latency P95 < 120s under normal conditions.
|
||||
- Data loss = 0 for outages shorter than 30 minutes (with available disk).
|
||||
- Collector liveness heartbeat every `pollSeconds * 3` max.
|
||||
|
||||
**Process**
|
||||
- Define SLI dashboards.
|
||||
- Define release gates tied to error budget burn.
|
||||
- Freeze risky changes when budget exhausted.
|
||||
|
||||
---
|
||||
|
||||
## Suggested implementation order inside repository
|
||||
|
||||
1. `file-operations-collector.ps1`: queue + sender + metrics (simpler flow).
|
||||
2. `dlp-endpoint-signals-collector.ps1`: queue + sender + metrics.
|
||||
3. Shared helper module extraction (`windows/lib/aw-transport.psm1`) for queue, retry, breaker.
|
||||
4. Policy cache and validation.
|
||||
5. Chaos test scripts and runbook.
|
||||
|
||||
---
|
||||
|
||||
## Deliverables checklist
|
||||
|
||||
- [ ] Transport queue implementation in both collectors.
|
||||
- [ ] Payload schema update with `eventId`.
|
||||
- [ ] Dedupe contract documented for server side.
|
||||
- [ ] Metrics fields added to heartbeat/logs.
|
||||
- [ ] Circuit breaker implemented.
|
||||
- [ ] Watcher auto-recreate implemented.
|
||||
- [ ] Last-known-good policy implemented.
|
||||
- [ ] Chaos test runbook and results.
|
||||
- [ ] SLO/error budget document adopted.
|
||||
@@ -0,0 +1,93 @@
|
||||
# Анализ DLP-скриптов PowerShell (работоспособность)
|
||||
|
||||
Дата анализа: **2026-05-04 (UTC)**
|
||||
|
||||
## Проверенный scope
|
||||
|
||||
- `windows/dlp-endpoint-signals-collector.ps1`
|
||||
- `windows/file-operations-collector.ps1`
|
||||
- `windows/dlp-policy.example.json`
|
||||
- `windows/web-category-rules.example.json`
|
||||
|
||||
## Ключевой итог
|
||||
|
||||
DLP-скрипты в целом рабочие по архитектуре (heartbeat в ActivityWatch, policy-driven правила, cooldown, enforcement), но есть **критичный риск misconfiguration** и несколько эксплуатационных рисков.
|
||||
|
||||
---
|
||||
|
||||
## Что точно хорошо
|
||||
|
||||
1. В обоих коллекторах включены `Set-StrictMode -Version Latest` и `$ErrorActionPreference = 'Stop'`.
|
||||
2. Есть отправка событий в отдельные bucket’ы (`aw-dlp-endpoint-signals_*`, `aw-dlp-incidents_*`, `aw-file-operations_*`).
|
||||
3. В endpoint-коллекторе реализованы:
|
||||
- правила по буферу обмена / USB / печати,
|
||||
- suppression через cooldown (`Should-EmitByCooldown`),
|
||||
- опциональный screenshot capture при инциденте.
|
||||
4. В file collector есть наблюдение за `Desktop/Documents/Downloads` через `FileSystemWatcher`.
|
||||
|
||||
---
|
||||
|
||||
## Найденные проблемы и риски
|
||||
|
||||
### 1) Критично: дефолтный путь конфига в endpoint-скрипте не совпадает с проектом
|
||||
|
||||
- `dlp-endpoint-signals-collector.ps1` использует по умолчанию:
|
||||
- `C:\ProgramData\ActivityWatch\deployment-config.json`
|
||||
- Остальной проект использует namespace `AWatch-rus` (`C:\ProgramData\AWatch-rus\...`).
|
||||
|
||||
**Риск:** endpoint-коллектор может стартовать без нужного deployment-конфига и работать с неверными/пустыми параметрами.
|
||||
|
||||
### 2) Нет строгой проверки HTTP-результата в file collector
|
||||
|
||||
В `file-operations-collector.ps1` POST выполняется через `HttpClient`, но код ответа не валидируется (`IsSuccessStatusCode` не проверяется), ошибки частично только логируются.
|
||||
|
||||
**Риск:** «тихая» потеря telemetry при 4xx/5xx.
|
||||
|
||||
### 3) Watcher не снимает event subscriptions явно
|
||||
|
||||
Есть `Register-ObjectEvent`, но в `finally` disposal только watcher-объектов; отписка событий (`Unregister-Event`) явно не делается.
|
||||
|
||||
**Риск:** при рестартах/долгой работе возможно накопление подписок в сессии.
|
||||
|
||||
### 4) Screenshot/GUI-зависимость для enforcement
|
||||
|
||||
`Capture-IncidentScreenshot` и balloon notification завязаны на `System.Windows.Forms/System.Drawing`.
|
||||
|
||||
**Риск:** в non-interactive / service context часть enforcement UX может не работать (событие уйдёт, но скриншот/уведомление может не сформироваться).
|
||||
|
||||
---
|
||||
|
||||
## Рекомендации (приоритет)
|
||||
|
||||
1. **P1:** выровнять дефолтный `ConfigPath` в `dlp-endpoint-signals-collector.ps1` на `C:\ProgramData\AWatch-rus\deployment-config.json`.
|
||||
2. **P1:** добавить проверку `response.IsSuccessStatusCode` в `file-operations-collector.ps1` и логировать body/status при ошибках.
|
||||
3. **P2:** сохранить subscription-объекты `Register-ObjectEvent` и делать `Unregister-Event` в `finally`.
|
||||
4. **P2:** для enforcement/UI добавить fallback режим «headless» (только лог + heartbeat).
|
||||
|
||||
---
|
||||
|
||||
## Что не удалось проверить в текущей среде
|
||||
|
||||
В этом контейнере отсутствует `pwsh`, поэтому не выполнены:
|
||||
|
||||
- синтаксический parse всех `*.ps1/*.psm1` через PowerShell parser;
|
||||
- `Test-ModuleManifest`;
|
||||
- smoke-run на Windows API (`Get-WinEvent`, `Get-Partition`, `Get-Disk`, `Set-Clipboard`, `Win32_PrintJob`).
|
||||
|
||||
---
|
||||
|
||||
## Команды для целевой Windows-проверки
|
||||
|
||||
```powershell
|
||||
# 1) Синтаксис
|
||||
Get-ChildItem .\windows -Recurse -Include *.ps1,*.psm1 | ForEach-Object {
|
||||
[void][System.Management.Automation.Language.Parser]::ParseFile($_.FullName,[ref]$null,[ref]$errs)
|
||||
if($errs){ "FAIL $($_.FullName)" } else { "OK $($_.FullName)" }
|
||||
}
|
||||
|
||||
# 2) Быстрый запуск file collector (с логом)
|
||||
.\windows\file-operations-collector.ps1 -ConfigPath 'C:\ProgramData\AWatch-rus\deployment-config.json' -LogPath 'C:\ProgramData\AWatch-rus\collector-fileops.log'
|
||||
|
||||
# 3) Быстрый запуск endpoint collector (с логом)
|
||||
.\windows\dlp-endpoint-signals-collector.ps1 -ConfigPath 'C:\ProgramData\AWatch-rus\deployment-config.json' -PolicyPath 'C:\ProgramData\AWatch-rus\dlp-policy.json' -LogPath 'C:\ProgramData\AWatch-rus\collector-endpoint.log'
|
||||
```
|
||||
Reference in New Issue
Block a user