DLP
This commit is contained in:
@@ -0,0 +1,52 @@
|
|||||||
|
AGENTS for OpenCode
|
||||||
|
|
||||||
|
Keep this file minimal and high-signal: only include facts an agent would otherwise miss.
|
||||||
|
|
||||||
|
1) Repo purpose (one line)
|
||||||
|
- This repository bundles an ActivityWatch Server deployment, RU WebUI patch, Windows collectors (PowerShell), and small Python utilities for aggregation and monitoring.
|
||||||
|
|
||||||
|
2) Highest-value entrypoints & commands
|
||||||
|
- Read README.md and docs/preparation.md first (they are the authoritative onboarding flow).
|
||||||
|
- Create a Proxmox CT: `proxmox/create-ct.sh [ /path/to/deploy.secrets.env ]` (reads secrets/deploy.secrets.env).
|
||||||
|
- Push server artifacts to an existing CT: `proxmox/push-aw-artifacts.sh [ /path/to/deploy.secrets.env ]`.
|
||||||
|
- Install AW server on the CT (runs inside CT): `aw-server/install_aw_server.sh` (requires `/etc/activitywatch/aw-server.env`).
|
||||||
|
- Apply RU WebUI patch (must run on the CT and after webui is present): `aw-server/apply_webui_ru_patch.sh`.
|
||||||
|
- Run the Windows ensemble deploy from a Windows admin host: `windows/deploy-ensemble.ps1` (see its parameters; it calls `deploy-domain-users.ps1`).
|
||||||
|
- Quick DLP aggregation (local): `python3 scripts/aggregate_dlp_events.py`.
|
||||||
|
|
||||||
|
3) Exact env/secrets behavior agents often miss
|
||||||
|
- Secrets live in `secrets/deploy.secrets.env` (actual file is intentionally local-only). Many scripts default to that path if no arg provided. Never add real secrets to commits. Use `.example` files as templates.
|
||||||
|
- The CT bootstrap workflow expects `/etc/activitywatch/aw-server.env` on the CT (pushed by push-aw-artifacts when AW_SERVER_* variables are set). `install_aw_server.sh` sources that exact path.
|
||||||
|
|
||||||
|
4) CI / quality checks the repo enforces
|
||||||
|
- GitHub CI runs shellcheck for `*.sh` and PSScriptAnalyzer for PowerShell in `windows/*.ps1` (see .github/workflows/ci.yml).
|
||||||
|
- Local quality gate: `scripts/quality-gate.sh` performs bash `-n`, (optional) shellcheck, pwsh parse checks, and ansible syntax checks. Run this before PRs.
|
||||||
|
|
||||||
|
5) File locations and toolchain quirks
|
||||||
|
- AW server binary is installed under `/opt/activitywatch/releases` and symlinked from `/opt/activitywatch/bin/aw-server-rust` by `install_aw_server.sh`.
|
||||||
|
- RU WebUI patching expects JS assets in `$AW_SERVER_WEBUI_DIR` (default `/opt/activitywatch/webui-ru`). The patch script writes `js/ru-patch-v5.js` and edits `index.html` in-place (it makes backups with .bak timestamps).
|
||||||
|
- `proxmox/create-ct.sh` and `proxmox/push-aw-artifacts.sh` source the same deploy.secrets.env and require many CT_* / AW_SERVER_* variables; missing vars cause immediate exit.
|
||||||
|
|
||||||
|
6) Monorepo boundaries / responsibilities
|
||||||
|
- aw-server/: server install & systemd unit + RU webui patching.
|
||||||
|
- proxmox/: create CT and push artifacts scripts (requires Proxmox `pct` CLI and CT preconditions).
|
||||||
|
- windows/: PowerShell collectors and deployment automation (Target: Windows admin hosts; validated via `validate-deployment.ps1`).
|
||||||
|
- grafana-1c/, pfsense/: monitoring stacks and pollers (separate deploys, not part of aw-server install).
|
||||||
|
- scripts/: small utilities and `quality-gate.sh` used by contributors.
|
||||||
|
|
||||||
|
7) Common gotchas
|
||||||
|
- Do NOT commit secrets (secrets/ are local-only; PRs must not contain real secrets).
|
||||||
|
- Many scripts assume they run on the target CT or on a Linux admin host with `pct` available. Don't try to run them on macOS without adapting dependencies.
|
||||||
|
- `aw-server/install_aw_server.sh` expects network access to download the AW release URL provided by AW_SERVER_DOWNLOAD_URL.
|
||||||
|
- `aw-server/apply_webui_ru_patch.sh` must run after AW webui files are present; it will fail if required bootstrap files under `/root/bootstrap` are missing.
|
||||||
|
- When pushing AW_SERVER env via `push-aw-artifacts.sh` the script will only write `/etc/activitywatch/aw-server.env` if all AW_SERVER_* variables are set; otherwise it warns and skips.
|
||||||
|
|
||||||
|
8) PR / commit checklist for agents
|
||||||
|
- Run `scripts/quality-gate.sh` locally (or ensure CI covers changed files).
|
||||||
|
- Ensure no secrets (.env with real values) are staged.
|
||||||
|
- If changing PowerShell, ensure PSScriptAnalyzer rules pass (CI enforces this).
|
||||||
|
|
||||||
|
9) Where to find more instructions (preserve these files)
|
||||||
|
- README.md, docs/preparation.md, docs/deployment.md, docs/runbook.md, docs/operations.md, docs/codebase-onboarding.md — read these when doing infra or deployment work.
|
||||||
|
|
||||||
|
If you need me to add step-by-step repros or automate one of the tasks above (create CT, push artifacts, run install on CT), say which one and I will implement the helper or run the checked commands.
|
||||||
@@ -14,6 +14,7 @@
|
|||||||
- `ansible/provision_proxmox_ct_matrix_and_deploy_aw.yml` — массовый полный playbook (несколько CT).
|
- `ansible/provision_proxmox_ct_matrix_and_deploy_aw.yml` — массовый полный playbook (несколько CT).
|
||||||
- `ansible/deploy_aw_windows.yml` — WinRM playbook для развёртывания Windows/RDP collector'ов.
|
- `ansible/deploy_aw_windows.yml` — WinRM playbook для развёртывания Windows/RDP collector'ов.
|
||||||
- `ansible/deploy_aw_pfsense_poller.yml` — развёртывание pfSense poller'а.
|
- `ansible/deploy_aw_pfsense_poller.yml` — развёртывание pfSense poller'а.
|
||||||
|
- `ansible/deploy_tsj_guardian_bot_proxmox.yml` — развёртывание TSJ Guardian Telegram Bot на Proxmox host.
|
||||||
- `ansible/install_full_stack.yml` — полный установочный playbook (оркестратор всех этапов).
|
- `ansible/install_full_stack.yml` — полный установочный playbook (оркестратор всех этапов).
|
||||||
- `ansible/inventory.example.ini` — шаблон inventory.
|
- `ansible/inventory.example.ini` — шаблон inventory.
|
||||||
- `ansible/group_vars/*.example.yml` — шаблоны переменных.
|
- `ansible/group_vars/*.example.yml` — шаблоны переменных.
|
||||||
@@ -161,6 +162,22 @@ Playbook:
|
|||||||
- пишет `/etc/aw-pfsense/poller.json`;
|
- пишет `/etc/aw-pfsense/poller.json`;
|
||||||
- поднимает `aw-pfsense-poller.service`.
|
- поднимает `aw-pfsense-poller.service`.
|
||||||
|
|
||||||
|
## Развёртывание TSJ Guardian Bot на Proxmox
|
||||||
|
|
||||||
|
1. Подготовьте vars:
|
||||||
|
- `cp ansible/group_vars/proxmox-bot.example.yml ansible/group_vars/proxmox-bot.yml`
|
||||||
|
2. Заполните минимум:
|
||||||
|
- `telegram_bot_token`
|
||||||
|
- `telegram_allowed_chat_ids`
|
||||||
|
- `tsj_bot_source_local_path`
|
||||||
|
3. Убедитесь, что в inventory есть группа `[proxmox]`.
|
||||||
|
4. Запустите:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ansible
|
||||||
|
ansible-playbook -i inventory.ini deploy_tsj_guardian_bot_proxmox.yml
|
||||||
|
```
|
||||||
|
|
||||||
## Результат
|
## Результат
|
||||||
|
|
||||||
- Установлен ActivityWatch Server.
|
- Установлен ActivityWatch Server.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
[aw_server]
|
[aw_server]
|
||||||
localhost ansible_connection=local ansible_user=root
|
aw-prod ansible_host=10.10.10.13 ansible_user=igor ansible_password=04091968 ansible_connection=ssh ansible_become=true ansible_become_method=sudo ansible_become_password=04091968
|
||||||
|
|
||||||
[aw_windows]
|
[aw_windows]
|
||||||
rdp-prod ansible_host=192.168.100.21 ansible_user=Администратор ansible_password=Sergei2009@ ansible_connection=winrm ansible_winrm_transport=ntlm ansible_port=5985 ansible_winrm_server_cert_validation=ignore
|
rdp-prod ansible_host=192.168.100.21 ansible_user=Администратор ansible_password=Sergei2009@ ansible_connection=winrm ansible_winrm_transport=ntlm ansible_port=5985 ansible_winrm_server_cert_validation=ignore
|
||||||
|
|||||||
@@ -35,3 +35,19 @@ def validate_snils(value: str) -> bool:
|
|||||||
if expected == 100:
|
if expected == 100:
|
||||||
expected = 0
|
expected = 0
|
||||||
return checksum == expected
|
return checksum == expected
|
||||||
|
|
||||||
|
|
||||||
|
def validate_passport(value: str) -> bool:
|
||||||
|
"""
|
||||||
|
Lightweight Russian passport validator:
|
||||||
|
- expects 10 digits (series+number), optionally with spaces
|
||||||
|
- rejects obvious invalid placeholders (all same digit, all zeros)
|
||||||
|
"""
|
||||||
|
digits = re.sub(r"\D", "", value)
|
||||||
|
if len(digits) != 10:
|
||||||
|
return False
|
||||||
|
if digits == "0000000000":
|
||||||
|
return False
|
||||||
|
if len(set(digits)) == 1:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|||||||
@@ -10,8 +10,8 @@
|
|||||||
"description": "СНИЛС"
|
"description": "СНИЛС"
|
||||||
},
|
},
|
||||||
"passport": {
|
"passport": {
|
||||||
"regex": "\\b\\d{4}\\s?\\d{6}\\b",
|
"regex": "\\b\\d{4}\\s\\d{6}\\b",
|
||||||
"checksum": "none",
|
"checksum": "passport",
|
||||||
"description": "Паспорт РФ"
|
"description": "Паспорт РФ"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,9 +4,11 @@ from __future__ import annotations
|
|||||||
import json
|
import json
|
||||||
import pathlib
|
import pathlib
|
||||||
import re
|
import re
|
||||||
|
import sys
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from checksum_validator import validate_inn, validate_snils
|
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||||
|
from checksum_validator import validate_inn, validate_passport, validate_snils
|
||||||
|
|
||||||
|
|
||||||
def _validate(kind: str, value: str) -> bool:
|
def _validate(kind: str, value: str) -> bool:
|
||||||
@@ -14,11 +16,17 @@ def _validate(kind: str, value: str) -> bool:
|
|||||||
return validate_inn(value)
|
return validate_inn(value)
|
||||||
if kind == "snils":
|
if kind == "snils":
|
||||||
return validate_snils(value)
|
return validate_snils(value)
|
||||||
|
if kind == "passport":
|
||||||
|
return validate_passport(value)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
def match_text(text: str, dictionary_path: str) -> list[dict[str, Any]]:
|
def _load_json(path: str) -> dict[str, Any]:
|
||||||
rules = json.loads(pathlib.Path(dictionary_path).read_text(encoding="utf-8"))
|
return json.loads(pathlib.Path(path).read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def match_text_with_dictionary(text: str, dictionary_path: str) -> list[dict[str, Any]]:
|
||||||
|
rules = _load_json(dictionary_path)
|
||||||
results: list[dict[str, Any]] = []
|
results: list[dict[str, Any]] = []
|
||||||
for name, rule in rules.items():
|
for name, rule in rules.items():
|
||||||
regex = re.compile(rule["regex"])
|
regex = re.compile(rule["regex"])
|
||||||
@@ -36,3 +44,40 @@ def match_text(text: str, dictionary_path: str) -> list[dict[str, Any]]:
|
|||||||
}
|
}
|
||||||
)
|
)
|
||||||
return results
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def match_text_with_regex_pack(text: str, regex_pack_path: str) -> list[dict[str, Any]]:
|
||||||
|
pack = _load_json(regex_pack_path)
|
||||||
|
results: list[dict[str, Any]] = []
|
||||||
|
entries: list[dict[str, Any]] = []
|
||||||
|
if isinstance(pack.get("rules"), list):
|
||||||
|
entries = [e for e in pack["rules"] if isinstance(e, dict)]
|
||||||
|
elif isinstance(pack.get("patterns"), dict):
|
||||||
|
entries = [{"id": k, **v} for k, v in pack["patterns"].items() if isinstance(v, dict)]
|
||||||
|
|
||||||
|
for entry in entries:
|
||||||
|
rule_id = entry.get("id") or entry.get("name") or "regex-rule"
|
||||||
|
regex = re.compile(entry["regex"])
|
||||||
|
for m in regex.finditer(text):
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"name": rule_id,
|
||||||
|
"description": entry.get("description", rule_id),
|
||||||
|
"value": m.group(0),
|
||||||
|
"start": m.start(),
|
||||||
|
"end": m.end(),
|
||||||
|
"severity": entry.get("severity", "medium"),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def match_text(
|
||||||
|
text: str,
|
||||||
|
dictionary_path: str | None = None,
|
||||||
|
regex_pack_path: str | None = None,
|
||||||
|
) -> dict[str, list[dict[str, Any]]]:
|
||||||
|
return {
|
||||||
|
"dictionary_matches": match_text_with_dictionary(text, dictionary_path) if dictionary_path else [],
|
||||||
|
"regex_matches": match_text_with_regex_pack(text, regex_pack_path) if regex_pack_path else [],
|
||||||
|
}
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ from pathlib import Path
|
|||||||
from PIL import Image
|
from PIL import Image
|
||||||
import pytesseract
|
import pytesseract
|
||||||
|
|
||||||
|
from dictionary_matcher import match_text
|
||||||
|
|
||||||
|
|
||||||
def extract_text(image_path: str) -> str:
|
def extract_text(image_path: str) -> str:
|
||||||
path = Path(image_path)
|
path = Path(image_path)
|
||||||
@@ -13,3 +15,16 @@ def extract_text(image_path: str) -> str:
|
|||||||
return ""
|
return ""
|
||||||
img = Image.open(path)
|
img = Image.open(path)
|
||||||
return pytesseract.image_to_string(img, lang="rus+eng")
|
return pytesseract.image_to_string(img, lang="rus+eng")
|
||||||
|
|
||||||
|
|
||||||
|
def analyze_screenshot(
|
||||||
|
image_path: str,
|
||||||
|
dictionary_path: str | None = None,
|
||||||
|
regex_pack_path: str | None = None,
|
||||||
|
) -> dict:
|
||||||
|
text = extract_text(image_path)
|
||||||
|
if not text:
|
||||||
|
return {"text": "", "dictionary_matches": [], "regex_matches": []}
|
||||||
|
result = match_text(text=text, dictionary_path=dictionary_path, regex_pack_path=regex_pack_path)
|
||||||
|
result["text"] = text
|
||||||
|
return result
|
||||||
|
|||||||
@@ -10,6 +10,8 @@
|
|||||||
- Print job мониторинг
|
- Print job мониторинг
|
||||||
- USB write блокировка
|
- USB write блокировка
|
||||||
- DLP правило evaluation
|
- DLP правило evaluation
|
||||||
|
- Advanced Content Analysis (dictionaryPack/regexPack)
|
||||||
|
- Checksum валидация ПДн (ИНН/СНИЛС/паспорт)
|
||||||
- Скриншоты при инцидентах
|
- Скриншоты при инцидентах
|
||||||
|
|
||||||
**Файл:** `windows/dlp-endpoint-signals-collector.ps1`
|
**Файл:** `windows/dlp-endpoint-signals-collector.ps1`
|
||||||
@@ -54,6 +56,16 @@
|
|||||||
**Технологии:** Rust
|
**Технологии:** Rust
|
||||||
**API:** HTTP на порту 5600
|
**API:** HTTP на порту 5600
|
||||||
|
|
||||||
|
### DLP Content Analysis
|
||||||
|
Серверные модули для анализа текста и OCR.
|
||||||
|
|
||||||
|
**Функции:**
|
||||||
|
- Словари ПДн 152-ФЗ
|
||||||
|
- Regex packs (financial/contacts/secrets)
|
||||||
|
- OCR распознавание скриншотов
|
||||||
|
|
||||||
|
**Файлы:** `aw-server/dlp-content-analysis/*`
|
||||||
|
|
||||||
### PostgreSQL Database
|
### PostgreSQL Database
|
||||||
Основное хранилище данных.
|
Основное хранилище данных.
|
||||||
|
|
||||||
|
|||||||
+114
-10
@@ -146,6 +146,10 @@ def _latest_bucket_ts(api_base: str, bucket_id: str, bucket_meta: dict[str, Any]
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _bucket_suffix(bucket_id: str, prefix: str) -> str:
|
||||||
|
return bucket_id[len(prefix):] if bucket_id.startswith(prefix) else bucket_id
|
||||||
|
|
||||||
|
|
||||||
def check_bucket_group(
|
def check_bucket_group(
|
||||||
report: HealthReport,
|
report: HealthReport,
|
||||||
api_base: str,
|
api_base: str,
|
||||||
@@ -202,6 +206,115 @@ def check_bucket_group(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _worktime_activity_map(api_base: str, buckets: dict[str, Any], max_age_seconds: int) -> dict[str, dict[str, Any]]:
|
||||||
|
now = _now_utc()
|
||||||
|
activity: dict[str, dict[str, Any]] = {}
|
||||||
|
prefix = "aw-worktime-sessions_"
|
||||||
|
for bucket_id in sorted(key for key in buckets if key.startswith(prefix)):
|
||||||
|
host = _bucket_suffix(bucket_id, prefix)
|
||||||
|
latest_ts: datetime | None = None
|
||||||
|
latest_active = False
|
||||||
|
try:
|
||||||
|
events = _http_json(f"{api_base}/buckets/{bucket_id}/events?limit=20")
|
||||||
|
except Exception:
|
||||||
|
activity[host] = {"active": False, "age_seconds": None, "bucket": bucket_id}
|
||||||
|
continue
|
||||||
|
if isinstance(events, list):
|
||||||
|
for event in events:
|
||||||
|
ts = _parse_ts(event.get("timestamp"))
|
||||||
|
if ts is None:
|
||||||
|
continue
|
||||||
|
if latest_ts is None or ts > latest_ts:
|
||||||
|
latest_ts = ts
|
||||||
|
latest_active = bool((event.get("data") or {}).get("active"))
|
||||||
|
activity[host] = {
|
||||||
|
"active": bool(latest_ts and latest_active and (_age_seconds(latest_ts, now) or 0) <= max_age_seconds),
|
||||||
|
"age_seconds": _age_seconds(latest_ts, now),
|
||||||
|
"bucket": bucket_id,
|
||||||
|
}
|
||||||
|
return activity
|
||||||
|
|
||||||
|
|
||||||
|
def check_file_operations_buckets(
|
||||||
|
report: HealthReport,
|
||||||
|
api_base: str,
|
||||||
|
buckets: dict[str, Any],
|
||||||
|
max_age_seconds: int,
|
||||||
|
strict: bool,
|
||||||
|
) -> None:
|
||||||
|
now = _now_utc()
|
||||||
|
prefix = "aw-file-operations_"
|
||||||
|
matched = sorted(bucket_id for bucket_id in buckets if bucket_id.startswith(prefix))
|
||||||
|
worktime = _worktime_activity_map(api_base, buckets, max_age_seconds)
|
||||||
|
active_hosts = sorted(host for host, meta in worktime.items() if meta.get("active"))
|
||||||
|
matched_by_host = {_bucket_suffix(bucket_id, prefix): bucket_id for bucket_id in matched}
|
||||||
|
|
||||||
|
ignored_unmanaged: list[str] = []
|
||||||
|
ignored_inactive: list[str] = []
|
||||||
|
missing_active: list[str] = []
|
||||||
|
stale: list[dict[str, Any]] = []
|
||||||
|
unknown: list[str] = []
|
||||||
|
fresh: list[str] = []
|
||||||
|
|
||||||
|
for host, bucket_id in matched_by_host.items():
|
||||||
|
if host not in worktime:
|
||||||
|
ignored_unmanaged.append(bucket_id)
|
||||||
|
continue
|
||||||
|
if host not in active_hosts:
|
||||||
|
ignored_inactive.append(bucket_id)
|
||||||
|
continue
|
||||||
|
ts = _latest_bucket_ts(api_base, bucket_id, buckets.get(bucket_id, {}))
|
||||||
|
age = _age_seconds(ts, now)
|
||||||
|
if age is None:
|
||||||
|
unknown.append(bucket_id)
|
||||||
|
continue
|
||||||
|
if age > max_age_seconds:
|
||||||
|
stale.append({"bucket": bucket_id, "age_seconds": age})
|
||||||
|
else:
|
||||||
|
fresh.append(bucket_id)
|
||||||
|
|
||||||
|
for host in active_hosts:
|
||||||
|
if host not in matched_by_host:
|
||||||
|
missing_active.append(host)
|
||||||
|
|
||||||
|
if not active_hosts:
|
||||||
|
report.add(
|
||||||
|
"buckets:file-operations",
|
||||||
|
"ok",
|
||||||
|
"no active managed hosts require file-operations freshness",
|
||||||
|
active_hosts=[],
|
||||||
|
ignored_unmanaged=ignored_unmanaged,
|
||||||
|
ignored_inactive=ignored_inactive,
|
||||||
|
worktime_hosts=sorted(worktime),
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
status = "ok"
|
||||||
|
summary = f"{len(fresh)} active host buckets fresh"
|
||||||
|
if missing_active:
|
||||||
|
status = "fail" if strict else "warn"
|
||||||
|
summary = f"{len(missing_active)} active hosts missing file-operations buckets"
|
||||||
|
elif stale:
|
||||||
|
status = "fail" if strict else "warn"
|
||||||
|
summary = f"{len(stale)} active host buckets stale"
|
||||||
|
elif unknown:
|
||||||
|
status = "warn"
|
||||||
|
summary = f"{len(unknown)} active host buckets without timestamp"
|
||||||
|
|
||||||
|
report.add(
|
||||||
|
"buckets:file-operations",
|
||||||
|
status,
|
||||||
|
summary,
|
||||||
|
active_hosts=active_hosts,
|
||||||
|
fresh=fresh,
|
||||||
|
stale=stale,
|
||||||
|
missing_active=missing_active,
|
||||||
|
unknown=unknown,
|
||||||
|
ignored_unmanaged=ignored_unmanaged,
|
||||||
|
ignored_inactive=ignored_inactive,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def check_endpoint_self_test_metrics(report: HealthReport, api_base: str, buckets: dict[str, Any]) -> None:
|
def check_endpoint_self_test_metrics(report: HealthReport, api_base: str, buckets: dict[str, Any]) -> None:
|
||||||
missing: list[str] = []
|
missing: list[str] = []
|
||||||
expected = ("queueDepth", "eventsEnqueued", "eventsFlushed", "sendFailures")
|
expected = ("queueDepth", "eventsEnqueued", "eventsFlushed", "sendFailures")
|
||||||
@@ -287,16 +400,7 @@ def main() -> int:
|
|||||||
raise RuntimeError("bucket list is not a dict")
|
raise RuntimeError("bucket list is not a dict")
|
||||||
report.add("aw:buckets-index", "ok", "bucket index loaded", total=len(buckets))
|
report.add("aw:buckets-index", "ok", "bucket index loaded", total=len(buckets))
|
||||||
check_bucket_group(report, aw_api_base, buckets, "endpoint-signals", "aw-dlp-endpoint-signals_", args.max_age_seconds)
|
check_bucket_group(report, aw_api_base, buckets, "endpoint-signals", "aw-dlp-endpoint-signals_", args.max_age_seconds)
|
||||||
check_bucket_group(
|
check_file_operations_buckets(report, aw_api_base, buckets, args.max_age_seconds, args.strict_fileops)
|
||||||
report,
|
|
||||||
aw_api_base,
|
|
||||||
buckets,
|
|
||||||
"file-operations",
|
|
||||||
"aw-file-operations_",
|
|
||||||
args.max_age_seconds,
|
|
||||||
severity_if_missing="warn",
|
|
||||||
severity_if_stale="fail" if args.strict_fileops else "warn",
|
|
||||||
)
|
|
||||||
check_bucket_group(report, aw_api_base, buckets, "incidents", "aw-dlp-incidents_", args.max_age_seconds * 24, severity_if_missing="warn", severity_if_stale="warn")
|
check_bucket_group(report, aw_api_base, buckets, "incidents", "aw-dlp-incidents_", args.max_age_seconds * 24, severity_if_missing="warn", severity_if_stale="warn")
|
||||||
check_endpoint_self_test_metrics(report, aw_api_base, buckets)
|
check_endpoint_self_test_metrics(report, aw_api_base, buckets)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
|
|||||||
@@ -587,6 +587,11 @@ function Load-DlpPolicy {
|
|||||||
usb = @()
|
usb = @()
|
||||||
print = @()
|
print = @()
|
||||||
}
|
}
|
||||||
|
contentAnalysis = [ordered]@{
|
||||||
|
dictionaryPack = $null
|
||||||
|
regexPack = $null
|
||||||
|
ocrEnabled = $false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
$script:PolicySource = 'defaults'
|
$script:PolicySource = 'defaults'
|
||||||
@@ -614,6 +619,18 @@ function Load-DlpPolicy {
|
|||||||
if ($props -contains 'usb' -and $raw.endpoint.usb) { $script:Policy.endpoint.usb = @($raw.endpoint.usb) }
|
if ($props -contains 'usb' -and $raw.endpoint.usb) { $script:Policy.endpoint.usb = @($raw.endpoint.usb) }
|
||||||
if ($props -contains 'print' -and $raw.endpoint.print) { $script:Policy.endpoint.print = @($raw.endpoint.print) }
|
if ($props -contains 'print' -and $raw.endpoint.print) { $script:Policy.endpoint.print = @($raw.endpoint.print) }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if ($raw.contentAnalysis) {
|
||||||
|
if ($raw.contentAnalysis.PSObject.Properties.Name -contains 'dictionaryPack' -and $raw.contentAnalysis.dictionaryPack) {
|
||||||
|
$script:Policy.contentAnalysis.dictionaryPack = [string]$raw.contentAnalysis.dictionaryPack
|
||||||
|
}
|
||||||
|
if ($raw.contentAnalysis.PSObject.Properties.Name -contains 'regexPack' -and $raw.contentAnalysis.regexPack) {
|
||||||
|
$script:Policy.contentAnalysis.regexPack = [string]$raw.contentAnalysis.regexPack
|
||||||
|
}
|
||||||
|
if ($raw.contentAnalysis.PSObject.Properties.Name -contains 'ocrEnabled') {
|
||||||
|
$script:Policy.contentAnalysis.ocrEnabled = [bool]$raw.contentAnalysis.ocrEnabled
|
||||||
|
}
|
||||||
|
}
|
||||||
$script:PolicySource = 'local'
|
$script:PolicySource = 'local'
|
||||||
}
|
}
|
||||||
catch {
|
catch {
|
||||||
@@ -621,6 +638,118 @@ function Load-DlpPolicy {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function Test-ValidInn {
|
||||||
|
param([string]$Value)
|
||||||
|
$digits = ($Value -replace '\D', '')
|
||||||
|
if ($digits.Length -eq 10) {
|
||||||
|
$coef = @(2, 4, 10, 3, 5, 9, 4, 6, 8)
|
||||||
|
$sum = 0
|
||||||
|
for ($i = 0; $i -lt 9; $i++) { $sum += ([int][string]$digits[$i]) * $coef[$i] }
|
||||||
|
$chk = ($sum % 11) % 10
|
||||||
|
return $chk -eq ([int][string]$digits[9])
|
||||||
|
}
|
||||||
|
if ($digits.Length -eq 12) {
|
||||||
|
$c11 = @(7, 2, 4, 10, 3, 5, 9, 4, 6, 8)
|
||||||
|
$c12 = @(3, 7, 2, 4, 10, 3, 5, 9, 4, 6, 8)
|
||||||
|
$sum11 = 0
|
||||||
|
for ($i = 0; $i -lt 10; $i++) { $sum11 += ([int][string]$digits[$i]) * $c11[$i] }
|
||||||
|
$sum12 = 0
|
||||||
|
for ($i = 0; $i -lt 11; $i++) { $sum12 += ([int][string]$digits[$i]) * $c12[$i] }
|
||||||
|
return ((($sum11 % 11) % 10) -eq ([int][string]$digits[10])) -and ((($sum12 % 11) % 10) -eq ([int][string]$digits[11]))
|
||||||
|
}
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-ValidSnils {
|
||||||
|
param([string]$Value)
|
||||||
|
$digits = ($Value -replace '\D', '')
|
||||||
|
if ($digits.Length -ne 11) { return $false }
|
||||||
|
$num = $digits.Substring(0, 9)
|
||||||
|
$checksum = [int]$digits.Substring(9, 2)
|
||||||
|
$sum = 0
|
||||||
|
for ($i = 0; $i -lt 9; $i++) { $sum += ([int][string]$num[$i]) * (9 - $i) }
|
||||||
|
if ($sum -lt 100) { $expected = $sum }
|
||||||
|
elseif ($sum -eq 100 -or $sum -eq 101) { $expected = 0 }
|
||||||
|
else {
|
||||||
|
$expected = $sum % 101
|
||||||
|
if ($expected -eq 100) { $expected = 0 }
|
||||||
|
}
|
||||||
|
return $checksum -eq $expected
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-ValidPassport {
|
||||||
|
param([string]$Value)
|
||||||
|
$digits = ($Value -replace '\D', '')
|
||||||
|
if ($digits.Length -ne 10) { return $false }
|
||||||
|
if ($digits -eq '0000000000') { return $false }
|
||||||
|
return ($digits.ToCharArray() | Select-Object -Unique).Count -gt 1
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-AdvancedContentMatches {
|
||||||
|
param(
|
||||||
|
[string]$Text,
|
||||||
|
[string]$DictionaryPack,
|
||||||
|
[string]$RegexPack
|
||||||
|
)
|
||||||
|
|
||||||
|
$result = @{
|
||||||
|
dictionaryMatches = @()
|
||||||
|
regexMatches = @()
|
||||||
|
}
|
||||||
|
if ([string]::IsNullOrWhiteSpace($Text)) { return $result }
|
||||||
|
|
||||||
|
if ($DictionaryPack -eq '152-fz-pdn') {
|
||||||
|
$m = [regex]::Matches($Text, '\b\d{10}\b|\b\d{12}\b')
|
||||||
|
foreach ($item in $m) {
|
||||||
|
if (Test-ValidInn -Value $item.Value) {
|
||||||
|
$result.dictionaryMatches += @{ name = 'inn'; value = $item.Value; severity = 'high' }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$m = [regex]::Matches($Text, '\b\d{3}-\d{3}-\d{3}\s?\d{2}\b')
|
||||||
|
foreach ($item in $m) {
|
||||||
|
if (Test-ValidSnils -Value $item.Value) {
|
||||||
|
$result.dictionaryMatches += @{ name = 'snils'; value = $item.Value; severity = 'high' }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$m = [regex]::Matches($Text, '\b\d{4}\s?\d{6}\b')
|
||||||
|
foreach ($item in $m) {
|
||||||
|
if (Test-ValidPassport -Value $item.Value) {
|
||||||
|
$result.dictionaryMatches += @{ name = 'passport'; value = $item.Value; severity = 'high' }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$regexRules = @()
|
||||||
|
switch ($RegexPack) {
|
||||||
|
'financial' {
|
||||||
|
$regexRules = @(
|
||||||
|
@{ id = 'card-pan'; regex = '\b(?:\d[ -]*?){13,19}\b'; severity = 'high' },
|
||||||
|
@{ id = 'iban'; regex = '\b[A-Z]{2}\d{2}[A-Z0-9]{11,30}\b'; severity = 'medium' }
|
||||||
|
)
|
||||||
|
}
|
||||||
|
'contacts' {
|
||||||
|
$regexRules = @(
|
||||||
|
@{ id = 'email'; regex = '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'; severity = 'low' },
|
||||||
|
@{ id = 'phone-ru'; regex = '(?:\+7|8)\s*\(?\d{3}\)?\s*\d{3}[- ]?\d{2}[- ]?\d{2}'; severity = 'low' }
|
||||||
|
)
|
||||||
|
}
|
||||||
|
'secrets' {
|
||||||
|
$regexRules = @(
|
||||||
|
@{ id = 'aws-access-key'; regex = 'AKIA[0-9A-Z]{16}'; severity = 'high' },
|
||||||
|
@{ id = 'generic-password'; regex = '(?i)(password|пароль)\s*[:=]\s*\S{6,}'; severity = 'medium' }
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
foreach ($rule in $regexRules) {
|
||||||
|
$m = [regex]::Matches($Text, [string]$rule.regex)
|
||||||
|
foreach ($item in $m) {
|
||||||
|
$result.regexMatches += @{ name = [string]$rule.id; value = $item.Value; severity = [string]$rule.severity }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return $result
|
||||||
|
}
|
||||||
|
|
||||||
function Apply-PolicyFromBundle {
|
function Apply-PolicyFromBundle {
|
||||||
param(
|
param(
|
||||||
[Parameter(Mandatory = $true)]$Bundle,
|
[Parameter(Mandatory = $true)]$Bundle,
|
||||||
@@ -749,6 +878,9 @@ function Evaluate-ClipboardRules {
|
|||||||
if (-not $ruleId) { continue }
|
if (-not $ruleId) { continue }
|
||||||
$minLength = if ($rule.minLength) { [int]$rule.minLength } else { 0 }
|
$minLength = if ($rule.minLength) { [int]$rule.minLength } else { 0 }
|
||||||
$regexPatterns = if ($rule.regexPatterns) { @($rule.regexPatterns) } else { @() }
|
$regexPatterns = if ($rule.regexPatterns) { @($rule.regexPatterns) } else { @() }
|
||||||
|
$dictionaryPack = if ($rule.dictionaryPack) { [string]$rule.dictionaryPack } elseif ($script:Policy.contentAnalysis.dictionaryPack) { [string]$script:Policy.contentAnalysis.dictionaryPack } else { $null }
|
||||||
|
$regexPack = if ($rule.regexPack) { [string]$rule.regexPack } elseif ($script:Policy.contentAnalysis.regexPack) { [string]$script:Policy.contentAnalysis.regexPack } else { $null }
|
||||||
|
$ocrEnabled = if ($rule.PSObject.Properties.Name -contains 'ocrEnabled') { [bool]$rule.ocrEnabled } else { [bool]$script:Policy.contentAnalysis.ocrEnabled }
|
||||||
if ($ClipboardText.Length -lt $minLength) { continue }
|
if ($ClipboardText.Length -lt $minLength) { continue }
|
||||||
|
|
||||||
$matched = $false
|
$matched = $false
|
||||||
@@ -758,6 +890,9 @@ function Evaluate-ClipboardRules {
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
$advanced = Get-AdvancedContentMatches -Text $ClipboardText -DictionaryPack $dictionaryPack -RegexPack $regexPack
|
||||||
|
$advancedMatched = (@($advanced.dictionaryMatches).Count -gt 0) -or (@($advanced.regexMatches).Count -gt 0)
|
||||||
|
if ($advancedMatched) { $matched = $true }
|
||||||
|
|
||||||
if (-not $matched) { continue }
|
if (-not $matched) { continue }
|
||||||
|
|
||||||
@@ -779,6 +914,11 @@ function Evaluate-ClipboardRules {
|
|||||||
clipboardHash = $ClipboardHash
|
clipboardHash = $ClipboardHash
|
||||||
clipboardLength = $ClipboardText.Length
|
clipboardLength = $ClipboardText.Length
|
||||||
enforced = $enforced
|
enforced = $enforced
|
||||||
|
dictionaryPack = $dictionaryPack
|
||||||
|
regexPack = $regexPack
|
||||||
|
dictionaryMatches = @($advanced.dictionaryMatches)
|
||||||
|
regexMatches = @($advanced.regexMatches)
|
||||||
|
ocrRequested = $ocrEnabled
|
||||||
}
|
}
|
||||||
Write-EndpointLog ("incident clipboard rule={0} action={1} severity={2} enforced={3}" -f $ruleId, $action, $severity, $enforced)
|
Write-EndpointLog ("incident clipboard rule={0} action={1} severity={2} enforced={3}" -f $ruleId, $action, $severity, $enforced)
|
||||||
}
|
}
|
||||||
@@ -839,6 +979,12 @@ function Evaluate-PrintRules {
|
|||||||
if ($rule.documentRegex) {
|
if ($rule.documentRegex) {
|
||||||
$match = $match -and ($DocumentName -match [string]$rule.documentRegex)
|
$match = $match -and ($DocumentName -match [string]$rule.documentRegex)
|
||||||
}
|
}
|
||||||
|
$dictionaryPack = if ($rule.dictionaryPack) { [string]$rule.dictionaryPack } elseif ($script:Policy.contentAnalysis.dictionaryPack) { [string]$script:Policy.contentAnalysis.dictionaryPack } else { $null }
|
||||||
|
$regexPack = if ($rule.regexPack) { [string]$rule.regexPack } elseif ($script:Policy.contentAnalysis.regexPack) { [string]$script:Policy.contentAnalysis.regexPack } else { $null }
|
||||||
|
$ocrEnabled = if ($rule.PSObject.Properties.Name -contains 'ocrEnabled') { [bool]$rule.ocrEnabled } else { [bool]$script:Policy.contentAnalysis.ocrEnabled }
|
||||||
|
$advanced = Get-AdvancedContentMatches -Text $DocumentName -DictionaryPack $dictionaryPack -RegexPack $regexPack
|
||||||
|
$advancedMatched = (@($advanced.dictionaryMatches).Count -gt 0) -or (@($advanced.regexMatches).Count -gt 0)
|
||||||
|
if ($advancedMatched) { $match = $true }
|
||||||
if (-not $match) { continue }
|
if (-not $match) { continue }
|
||||||
|
|
||||||
$cooldown = if ($rule.cooldownSeconds) { [int]$rule.cooldownSeconds } else { [int]$script:Policy.defaults.cooldownSeconds }
|
$cooldown = if ($rule.cooldownSeconds) { [int]$rule.cooldownSeconds } else { [int]$script:Policy.defaults.cooldownSeconds }
|
||||||
@@ -860,6 +1006,11 @@ function Evaluate-PrintRules {
|
|||||||
documentName = $DocumentName
|
documentName = $DocumentName
|
||||||
owner = $Owner
|
owner = $Owner
|
||||||
enforced = $enforced
|
enforced = $enforced
|
||||||
|
dictionaryPack = $dictionaryPack
|
||||||
|
regexPack = $regexPack
|
||||||
|
dictionaryMatches = @($advanced.dictionaryMatches)
|
||||||
|
regexMatches = @($advanced.regexMatches)
|
||||||
|
ocrRequested = $ocrEnabled
|
||||||
}
|
}
|
||||||
Write-EndpointLog ("incident print rule={0} action={1} severity={2} printer={3} enforced={4}" -f $ruleId, $action, $severity, $PrinterName, $enforced)
|
Write-EndpointLog ("incident print rule={0} action={1} severity={2} printer={3} enforced={4}" -f $ruleId, $action, $severity, $PrinterName, $enforced)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -96,6 +96,11 @@
|
|||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
"contentAnalysis": {
|
||||||
|
"dictionaryPack": "152-fz-pdn",
|
||||||
|
"regexPack": "secrets",
|
||||||
|
"ocrEnabled": true
|
||||||
|
},
|
||||||
"ioc": {
|
"ioc": {
|
||||||
"enabled": true,
|
"enabled": true,
|
||||||
"source": "http://10.10.10.13:5610/dlp-ioc/ioc_blacklist.json",
|
"source": "http://10.10.10.13:5610/dlp-ioc/ioc_blacklist.json",
|
||||||
|
|||||||
Reference in New Issue
Block a user