AWatch-rus/docs/dlp-reliability-roadmap.md

DLP Reliability Roadmap

Scope

Roadmap for improving runtime reliability of:

• windows/dlp-endpoint-signals-collector.ps1
• windows/file-operations-collector.ps1

Date: 2026-05-04

---

Stage 1 (1-2 days): Quick wins

1) Disk queue + sender loop + retry/backoff/jitter

Goal: no data loss on temporary network/server outages.

Tasks

• Add local append-only queue file per collector (*.jsonl) under ProgramData logs/artifacts root.
• Write events to queue first, then send asynchronously.
• Implement sender loop:
  • reads oldest unsent records,
  • sends in small batches,
  • marks sent records,
  • compacts queue periodically.
• Implement retry policy with exponential backoff + jitter.

Acceptance criteria

• When API is unavailable, queue grows and collector keeps running.
• When API recovers, queued events are flushed automatically.
• No collector crash during repeated network failures.

2) eventId + dedupe contract

Goal: at-least-once delivery without logical duplicates.

Tasks

• Add eventId (UUID), eventCreatedAt, collectorType, hostname to every payload.
• Define server dedupe contract:
  • dedupe key = eventId,
  • TTL for dedupe cache,
  • idempotent processing semantics.

Acceptance criteria

• Retried sends do not create duplicate incidents/events in downstream storage.
• Payload schema documentation updated.

3) Basic metrics/logging

Goal: visibility into health and data delivery.

Tasks

• Emit counters/gauges to log and heartbeat:
  • queueDepth,
  • oldestUnsentAgeSec,
  • eventsEnqueued,
  • eventsSent,
  • sendFailures,
  • lastSendStatus.

Acceptance criteria

• Operators can identify stuck queue and send failures from logs only.

---

Stage 2: Hardening

1) Circuit breaker + health probes

Tasks

• Add transport circuit breaker (Closed/Open/HalfOpen).
• Open breaker after N consecutive failures.
• In Open state perform probe every M seconds.
• Close breaker on successful probe.

Acceptance criteria

• Reduced request storm during outage.
• Deterministic recovery behavior after outage.

2) Watcher auto-recreate

Tasks

• Handle FileSystemWatcher error/overflow events.
• Recreate watcher and subscriptions automatically.
• Keep watchdog timer to ensure watcher health.

Acceptance criteria

• Watcher resumes after overflow without manual restart.

3) Last-known-good policy

Tasks

• Validate new policy before apply.
• Cache last valid policy with checksum/version.
• Rollback to cached policy on parse/validation errors.

Acceptance criteria

• Broken policy cannot stop detection loop.

---

Stage 3: Reliability operations

1) Chaos tests

Scenarios:

• network disconnect,
• API 5xx bursts,
• slow disk / queue write delay,
• headless UI context,
• forced collector restart.

Acceptance criteria

• For each scenario, documented expected behavior and observed result.
• No silent data loss in tested outage windows.

2) SLO + error budget process

Initial SLO proposals

• Event delivery latency P95 < 120s under normal conditions.
• Data loss = 0 for outages shorter than 30 minutes (with available disk).
• Collector liveness heartbeat every pollSeconds * 3 max.

Process

• Define SLI dashboards.
• Define release gates tied to error budget burn.
• Freeze risky changes when budget exhausted.

---

Suggested implementation order inside repository

1. file-operations-collector.ps1: queue + sender + metrics (simpler flow).
2. dlp-endpoint-signals-collector.ps1: queue + sender + metrics.
3. Shared helper module extraction (windows/lib/aw-transport.psm1) for queue, retry, breaker.
4. Policy cache and validation.
5. Chaos test scripts and runbook.

---

Deliverables checklist

• Transport queue implementation in both collectors.
• Payload schema update with eventId.
• Dedupe contract documented for server side.
• Metrics fields added to heartbeat/logs.
• Circuit breaker implemented.
• Watcher auto-recreate implemented.
• Last-known-good policy implemented.
• Chaos test runbook and results.
• SLO/error budget document adopted.