From e8bcf340790c7e092ddd726ffdc42ae29d9fe53e Mon Sep 17 00:00:00 2001
From: IgorRachkov <89467086+igor04091968@users.noreply.github.com>
Date: Mon, 4 May 2026 22:43:01 +0300
Subject: [PATCH] Add phased DLP reliability roadmap and acceptance criteria

---
 dlp-reliability-roadmap.md | 151 +++++++++++++++++++++++++++++++++++++
 powershell-analysis.md     |  93 +++++++++++++++++++++++
 2 files changed, 244 insertions(+)
 create mode 100644 dlp-reliability-roadmap.md
 create mode 100644 powershell-analysis.md

diff --git a/dlp-reliability-roadmap.md b/dlp-reliability-roadmap.md
new file mode 100644
index 0000000..6a93869
--- /dev/null
+++ b/dlp-reliability-roadmap.md
@@ -0,0 +1,151 @@
+# DLP Reliability Roadmap
+
+## Scope
+Roadmap for improving runtime reliability of:
+- `windows/dlp-endpoint-signals-collector.ps1`
+- `windows/file-operations-collector.ps1`
+
+Date: 2026-05-04
+
+---
+
+## Stage 1 (1-2 days): Quick wins
+
+### 1) Disk queue + sender loop + retry/backoff/jitter
+
+**Goal:** no data loss on temporary network/server outages.
+
+**Tasks**
+- Add local append-only queue file per collector (`*.jsonl`) under ProgramData logs/artifacts root.
+- Write events to queue first, then send asynchronously.
+- Implement sender loop:
+  - reads oldest unsent records,
+  - sends in small batches,
+  - marks sent records,
+  - compacts queue periodically.
+- Implement retry policy with exponential backoff + jitter.
+
+**Acceptance criteria**
+- When API is unavailable, queue grows and collector keeps running.
+- When API recovers, queued events are flushed automatically.
+- No collector crash during repeated network failures.
+
+### 2) `eventId` + dedupe contract
+
+**Goal:** at-least-once delivery without logical duplicates.
+
+**Tasks**
+- Add `eventId` (UUID), `eventCreatedAt`, `collectorType`, `hostname` to every payload.
+- Define server dedupe contract:
+  - dedupe key = `eventId`,
+  - TTL for dedupe cache,
+  - idempotent processing semantics.
+
+**Acceptance criteria**
+- Retried sends do not create duplicate incidents/events in downstream storage.
+- Payload schema documentation updated.
+
+### 3) Basic metrics/logging
+
+**Goal:** visibility into health and data delivery.
+
+**Tasks**
+- Emit counters/gauges to log and heartbeat:
+  - `queueDepth`,
+  - `oldestUnsentAgeSec`,
+  - `eventsEnqueued`,
+  - `eventsSent`,
+  - `sendFailures`,
+  - `lastSendStatus`.
+
+**Acceptance criteria**
+- Operators can identify stuck queue and send failures from logs only.
+
+---
+
+## Stage 2: Hardening
+
+### 1) Circuit breaker + health probes
+
+**Tasks**
+- Add transport circuit breaker (Closed/Open/HalfOpen).
+- Open breaker after N consecutive failures.
+- In Open state perform probe every M seconds.
+- Close breaker on successful probe.
+
+**Acceptance criteria**
+- Reduced request storm during outage.
+- Deterministic recovery behavior after outage.
+
+### 2) Watcher auto-recreate
+
+**Tasks**
+- Handle `FileSystemWatcher` error/overflow events.
+- Recreate watcher and subscriptions automatically.
+- Keep watchdog timer to ensure watcher health.
+
+**Acceptance criteria**
+- Watcher resumes after overflow without manual restart.
+
+### 3) Last-known-good policy
+
+**Tasks**
+- Validate new policy before apply.
+- Cache last valid policy with checksum/version.
+- Rollback to cached policy on parse/validation errors.
+
+**Acceptance criteria**
+- Broken policy cannot stop detection loop.
+
+---
+
+## Stage 3: Reliability operations
+
+### 1) Chaos tests
+
+Scenarios:
+- network disconnect,
+- API 5xx bursts,
+- slow disk / queue write delay,
+- headless UI context,
+- forced collector restart.
+
+**Acceptance criteria**
+- For each scenario, documented expected behavior and observed result.
+- No silent data loss in tested outage windows.
+
+### 2) SLO + error budget process
+
+**Initial SLO proposals**
+- Event delivery latency P95 < 120s under normal conditions.
+- Data loss = 0 for outages shorter than 30 minutes (with available disk).
+- Collector liveness heartbeat every `pollSeconds * 3` max.
+
+**Process**
+- Define SLI dashboards.
+- Define release gates tied to error budget burn.
+- Freeze risky changes when budget exhausted.
+
+---
+
+## Suggested implementation order inside repository
+
+1. `file-operations-collector.ps1`: queue + sender + metrics (simpler flow).
+2. `dlp-endpoint-signals-collector.ps1`: queue + sender + metrics.
+3. Shared helper module extraction (`windows/lib/aw-transport.psm1`) for queue, retry, breaker.
+4. Policy cache and validation.
+5. Chaos test scripts and runbook.
+
+---
+
+## Deliverables checklist
+
+- [ ] Transport queue implementation in both collectors.
+- [ ] Payload schema update with `eventId`.
+- [ ] Dedupe contract documented for server side.
+- [ ] Metrics fields added to heartbeat/logs.
+- [ ] Circuit breaker implemented.
+- [ ] Watcher auto-recreate implemented.
+- [ ] Last-known-good policy implemented.
+- [ ] Chaos test runbook and results.
+- [ ] SLO/error budget document adopted.
diff --git a/powershell-analysis.md b/powershell-analysis.md
new file mode 100644
index 0000000..d7af0ee
--- /dev/null
+++ b/powershell-analysis.md
@@ -0,0 +1,93 @@
+# Анализ DLP-скриптов PowerShell (работоспособность)
+
+Дата анализа: **2026-05-04 (UTC)**
+
+## Проверенный scope
+
+- `windows/dlp-endpoint-signals-collector.ps1`
+- `windows/file-operations-collector.ps1`
+- `windows/dlp-policy.example.json`
+- `windows/web-category-rules.example.json`
+
+## Ключевой итог
+
+DLP-скрипты в целом рабочие по архитектуре (heartbeat в ActivityWatch, policy-driven правила, cooldown, enforcement), но есть **критичный риск misconfiguration** и несколько эксплуатационных рисков.
+
+---
+
+## Что точно хорошо
+
+1. В обоих коллекторах включены `Set-StrictMode -Version Latest` и `$ErrorActionPreference = 'Stop'`.
+2. Есть отправка событий в отдельные bucket’ы (`aw-dlp-endpoint-signals_*`, `aw-dlp-incidents_*`, `aw-file-operations_*`).
+3. В endpoint-коллекторе реализованы:
+   - правила по буферу обмена / USB / печати,
+   - suppression через cooldown (`Should-EmitByCooldown`),
+   - опциональный screenshot capture при инциденте.
+4. В file collector есть наблюдение за `Desktop/Documents/Downloads` через `FileSystemWatcher`.
+
+---
+
+## Найденные проблемы и риски
+
+### 1) Критично: дефолтный путь конфига в endpoint-скрипте не совпадает с проектом
+
+- `dlp-endpoint-signals-collector.ps1` использует по умолчанию:
+  - `C:\ProgramData\ActivityWatch\deployment-config.json`
+- Остальной проект использует namespace `AWatch-rus` (`C:\ProgramData\AWatch-rus\...`).
+
+**Риск:** endpoint-коллектор может стартовать без нужного deployment-конфига и работать с неверными/пустыми параметрами.
+
+### 2) Нет строгой проверки HTTP-результата в file collector
+
+В `file-operations-collector.ps1` POST выполняется через `HttpClient`, но код ответа не валидируется (`IsSuccessStatusCode` не проверяется), ошибки частично только логируются.
+
+**Риск:** «тихая» потеря telemetry при 4xx/5xx.
+
+### 3) Watcher не снимает event subscriptions явно
+
+Есть `Register-ObjectEvent`, но в `finally` disposal только watcher-объектов; отписка событий (`Unregister-Event`) явно не делается.
+
+**Риск:** при рестартах/долгой работе возможно накопление подписок в сессии.
+
+### 4) Screenshot/GUI-зависимость для enforcement
+
+`Capture-IncidentScreenshot` и balloon notification завязаны на `System.Windows.Forms/System.Drawing`.
+
+**Риск:** в non-interactive / service context часть enforcement UX может не работать (событие уйдёт, но скриншот/уведомление может не сформироваться).
+
+---
+
+## Рекомендации (приоритет)
+
+1. **P1:** выровнять дефолтный `ConfigPath` в `dlp-endpoint-signals-collector.ps1` на `C:\ProgramData\AWatch-rus\deployment-config.json`.
+2. **P1:** добавить проверку `response.IsSuccessStatusCode` в `file-operations-collector.ps1` и логировать body/status при ошибках.
+3. **P2:** сохранить subscription-объекты `Register-ObjectEvent` и делать `Unregister-Event` в `finally`.
+4. **P2:** для enforcement/UI добавить fallback режим «headless» (только лог + heartbeat).
+
+---
+
+## Что не удалось проверить в текущей среде
+
+В этом контейнере отсутствует `pwsh`, поэтому не выполнены:
+
+- синтаксический parse всех `*.ps1/*.psm1` через PowerShell parser;
+- `Test-ModuleManifest`;
+- smoke-run на Windows API (`Get-WinEvent`, `Get-Partition`, `Get-Disk`, `Set-Clipboard`, `Win32_PrintJob`).
+
+---
+
+## Команды для целевой Windows-проверки
+
+```powershell
+# 1) Синтаксис
+Get-ChildItem .\windows -Recurse -Include *.ps1,*.psm1 | ForEach-Object {
+  [void][System.Management.Automation.Language.Parser]::ParseFile($_.FullName,[ref]$null,[ref]$errs)
+  if($errs){ "FAIL $($_.FullName)" } else { "OK $($_.FullName)" }
+}
+
+# 2) Быстрый запуск file collector (с логом)
+.\windows\file-operations-collector.ps1 -ConfigPath 'C:\ProgramData\AWatch-rus\deployment-config.json' -LogPath 'C:\ProgramData\AWatch-rus\collector-fileops.log'
+
+# 3) Быстрый запуск endpoint collector (с логом)
+.\windows\dlp-endpoint-signals-collector.ps1 -ConfigPath 'C:\ProgramData\AWatch-rus\deployment-config.json' -PolicyPath 'C:\ProgramData\AWatch-rus\dlp-policy.json' -LogPath 'C:\ProgramData\AWatch-rus\collector-endpoint.log'
+```