Compare commits

..
Author SHA1 Message Date
igor04091968 9c01a2f297 Add portal prewarm resilience docs
CI / Rust checks (push) Canceled after 0s
CI / Docs and registry checks (push) Canceled after 0s
CI / Smoke checks (push) Canceled after 0s
Coverage / Coverage baseline (push) Canceled after 0s
Security / Cargo audit (push) Canceled after 0s
Security / Cargo deny (push) Canceled after 0s
Security / Secret pattern check (push) Canceled after 0s
Security / Dependency review (push) Canceled after 0s
2026-07-01 06:13:06 +03:00
35 changed files with 1372 additions and 1947 deletions
+70
View File
@@ -1796,6 +1796,76 @@ systemctl is-active tsj-guardian-bot tsj-guardian-watchdog gost-tg
- требует laptop-only path или интерактивный shell;
- выводит секреты в stdout/journald/report.
## 12a. DetMir hardline resilience deltas
2026-06-24 Hayabusa poison-package isolation:
- `aw-hayabusa-autoprocess-rust` validates drop zip packages before `accept`;
- corrupt/empty/unsafe packages and invalid sidecars are quarantined under
`/opt/hayabusa/quarantine/drop/...` with `reason.json`;
- `aw-hayabusa process-inbox` isolates failed incoming packages under
`/opt/hayabusa/quarantine/incoming/...` and continues with the rest of the
queue;
- one poison zip no longer blocks the whole Hayabusa batch or trips systemd
start-limit by itself;
- verification: `bash -n aw-server/hayabusa/aw-hayabusa.sh` and
`cargo test -p hayabusa-tools` passed locally;
- live rollout on AW server completed on 2026-06-24 with backups, `doctor` OK,
isolated empty-drop dry-run OK, production `incoming_zip=0`, `DROP_COUNT=0`,
`staged_dirs=0`; stale 2026-06-20 staging residue moved to quarantine with
`reason.json`.
2026-06-24 Windows collector guard child watchdog:
- `AWatchRusCollectorGuardService.cs` now watches the child process via
`Process.Exited`;
- unexpected child exit triggers bounded restart inside the service wrapper;
- restart budget exhaustion exits the service with a non-zero code so SCM
recovery can restart the wrapper instead of leaving `running/no child`;
- `install-collector-guard-service.ps1` sets `sc.exe failureflag <service> 1`;
- runtime contract is unchanged: Rust collector guard remains the preferred
child, PowerShell guard remains fallback, and `ActivityWatch Recovery` remains
enabled as bootstrap fallback;
- local static verification passed: C# wrapper compiled through PowerShell
`Add-Type`; PowerShell installer parsed through the PowerShell parser;
- live rollout on RDP completed on 2026-06-24 with backup, reinstall in
`enforce` mode, `failureflag` enabled, controlled child-kill fault injection,
and final validation `service=Running`, `child_count=1`, latest Rust guard
cycle `status=ok`.
2026-06-24 DetMir contour resilience check:
- added `scripts/detmir_resilience_check.sh`;
- repo mode verifies hardening presence for Hayabusa poison quarantine, Windows
child watchdog, SCM `failureflag`, and resilience docs;
- live mode is read-only and checks local AW service/API state, failed systemd
units, Hayabusa queues/quarantine, and AW SQLite DB/WAL size thresholds;
- `scripts/run_awatch_contour_check.sh` can include it with
`RUN_RESILIENCE_CHECK=1` and `RESILIENCE_CHECK_MODE=repo|live|all`;
- strict secret mode (`DETMIR_RESILIENCE_STRICT_SECRETS=1`) fails literal
Ansible password assignments without printing secret values;
- local verification: shell syntax passed, repo mode passed with `ok=15`,
`fail=0`, and one WARN for literal private inventory password assignments;
- live AW server verification passed with `ok=9`, `fail=0`; one expected WARN
remains for the deliberate quarantine `reason.json` created during stale
staging cleanup.
2026-06-24 live healthd wrapper timeout correction:
- `aw-rus-healthd-rust` had a 20 second default wrapper timeout for
`/usr/local/bin/aw-health-check` and `/usr/local/bin/dlp-health-check --json`;
- under concurrent contour checks the DLP health command could exceed that
limit, be killed, and leave partial stdout that healthd reported as
`invalid JSON output`;
- production `/etc/activitywatch/aw-server.env` now sets
`AW_RUS_HEALTH_WRAPPER_TIMEOUT_SECONDS=90`, still below the service
`TimeoutStartSec=180`;
- `aw-server/aw-server.env.example` carries the same value so redeploys do not
restore the false-fail default;
- live verification after the change: `aw-rus-healthd.service` finished with
`status=0/SUCCESS`, `fail=0`, while AW API, Worktime API and DLP health were
independently reachable.
## 13. Рабочий принцип
Правильный перенос на Rust - это не переписывание строк один-в-один.
+2 -30
View File
@@ -43,9 +43,6 @@ struct Cli {
#[arg(long, default_value_t = false)]
no_color: bool,
#[arg(long, default_value_t = true)]
dlp_enabled: bool,
}
#[derive(Debug, Clone)]
@@ -112,12 +109,7 @@ fn main() {
}
fn run() -> Result<i32> {
let mut cli = Cli::parse();
if let Some(value) =
env_nonempty("AW_DLP_ENABLED").or_else(|| env_nonempty("DETMIR_DLP_ENABLED"))
{
cli.dlp_enabled = parse_env_flag(&value);
}
let cli = Cli::parse();
let server = cli
.server
.or_else(|| env_nonempty("AW_CHECK_SERVER"))
@@ -180,11 +172,7 @@ fn run() -> Result<i32> {
"---------------------------------------------", "--------", "----------------------"
);
for bucket in BUCKETS
.iter()
.copied()
.filter(|bucket| cli.dlp_enabled || !bucket.starts_with("aw-dlp-"))
{
for bucket in BUCKETS {
let bucket_full = format!("{bucket}_{host}");
let event = bucket_event(
&server,
@@ -202,15 +190,6 @@ fn run() -> Result<i32> {
render_status(&colors, status)
);
}
if !cli.dlp_enabled {
println!(
"{:<45} {:<8} {:<22} {}",
"aw-dlp-*",
"-",
"disabled",
colors.paint(colors.cyan, "SKIPPED")
);
}
println!();
println!("--- CORS Check ---");
@@ -514,13 +493,6 @@ fn env_nonempty(name: &str) -> Option<String> {
.filter(|value| !value.is_empty())
}
fn parse_env_flag(value: &str) -> bool {
matches!(
value.trim().to_ascii_lowercase().as_str(),
"1" | "true" | "yes" | "on"
)
}
#[cfg(test)]
mod tests {
use super::*;
+2 -16
View File
@@ -1,7 +1,6 @@
use std::fs::{self, File, OpenOptions};
use std::io::Write;
use std::os::unix::fs::symlink;
use std::os::unix::process::CommandExt;
use std::path::{Path, PathBuf};
use std::process::{Command, Output, Stdio};
use std::time::{Duration, SystemTime};
@@ -199,7 +198,6 @@ fn run_to_file(
let mut child = Command::new(command)
.args(args)
.process_group(0)
.stdout(Stdio::from(stdout))
.stderr(Stdio::from(stderr))
.spawn()
@@ -211,7 +209,7 @@ fn run_to_file(
break status.code().unwrap_or(1);
}
if started.elapsed() >= timeout {
terminate_process_group(child.id());
let _ = child.kill();
let _ = child.wait();
let mut stderr = OpenOptions::new().append(true).open(&stderr_path)?;
writeln!(
@@ -236,17 +234,6 @@ fn run_to_file(
Ok(rc)
}
fn terminate_process_group(child_pid: u32) {
let process_group = format!("-{child_pid}");
let _ = Command::new("/bin/kill")
.args(["-TERM", "--", &process_group])
.status();
std::thread::sleep(Duration::from_secs(2));
let _ = Command::new("/bin/kill")
.args(["-KILL", "--", &process_group])
.status();
}
fn read_rc(path: &Path) -> i32 {
fs::read_to_string(path)
.ok()
@@ -424,7 +411,6 @@ fn write_report(
fn run_report_command(polli_bin: &str, bundle: File, timeout: Duration) -> Result<Output> {
let mut child = Command::new(polli_bin)
.args(["--model", "text.daily", "--max-tokens", "900"])
.process_group(0)
.stdin(Stdio::from(bundle))
.stdout(Stdio::piped())
.stderr(Stdio::piped())
@@ -439,7 +425,7 @@ fn run_report_command(polli_bin: &str, bundle: File, timeout: Duration) -> Resul
.with_context(|| format!("failed to collect {polli_bin} output"));
}
if started.elapsed() >= timeout {
terminate_process_group(child.id());
let _ = child.kill();
let _ = child.wait();
anyhow::bail!(
"Pollinations report timed out after {} seconds",
+30 -133
View File
@@ -1,6 +1,5 @@
use std::io::Read;
use std::net::{SocketAddr, TcpStream};
use std::os::unix::process::CommandExt;
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
@@ -90,15 +89,9 @@ struct Cli {
#[arg(long, default_value_t = 45)]
dlp_timeout_seconds: u64,
#[arg(long, default_value_t = 150)]
overall_timeout_seconds: u64,
#[arg(long, default_value_t = false)]
disable_dlp_health_check: bool,
#[arg(long, default_value_t = true)]
dlp_enabled: bool,
#[arg(long, default_value_t = false)]
disable_portal_check: bool,
}
@@ -200,8 +193,8 @@ fn parse_env_flag(value: &str) -> bool {
)
}
fn bucket_specs(hostname: &str, dlp_enabled: bool) -> Vec<BucketSpec> {
let mut specs = vec![
fn bucket_specs(hostname: &str) -> Vec<BucketSpec> {
vec![
BucketSpec {
label: "AFK watcher",
bucket: format!("aw-watcher-afk_{hostname}"),
@@ -226,36 +219,31 @@ fn bucket_specs(hostname: &str, dlp_enabled: bool) -> Vec<BucketSpec> {
max_age_seconds: None,
mode: BucketMode::EventDriven,
},
];
if dlp_enabled {
specs.extend([
BucketSpec {
label: "DLP signals",
bucket: format!("aw-dlp-endpoint-signals_{hostname}"),
max_age_seconds: Some(10 * 60),
mode: BucketMode::InteractiveFresh,
},
BucketSpec {
label: "DLP incidents",
bucket: format!("aw-dlp-incidents_{hostname}"),
max_age_seconds: None,
mode: BucketMode::EventDriven,
},
BucketSpec {
label: "DLP review",
bucket: format!("aw-dlp-review_{hostname}"),
max_age_seconds: None,
mode: BucketMode::EventDriven,
},
BucketSpec {
label: "DLP rules",
bucket: format!("aw-dlp-rules_{hostname}"),
max_age_seconds: None,
mode: BucketMode::EventDriven,
},
]);
}
specs
BucketSpec {
label: "DLP signals",
bucket: format!("aw-dlp-endpoint-signals_{hostname}"),
max_age_seconds: Some(10 * 60),
mode: BucketMode::InteractiveFresh,
},
BucketSpec {
label: "DLP incidents",
bucket: format!("aw-dlp-incidents_{hostname}"),
max_age_seconds: None,
mode: BucketMode::EventDriven,
},
BucketSpec {
label: "DLP review",
bucket: format!("aw-dlp-review_{hostname}"),
max_age_seconds: None,
mode: BucketMode::EventDriven,
},
BucketSpec {
label: "DLP rules",
bucket: format!("aw-dlp-rules_{hostname}"),
max_age_seconds: None,
mode: BucketMode::EventDriven,
},
]
}
fn build_headers(items: &[(&str, &str)]) -> Result<HeaderMap> {
@@ -399,20 +387,7 @@ fn service_checks(args: &Cli) -> Vec<ServiceCheck> {
if security_events_clickhouse_enabled(args) {
checks.push(clickhouse_security_events_check(args));
}
if !args.dlp_enabled {
checks.push(ServiceCheck {
name: "aw-dlp-mode".to_string(),
required: false,
ok: true,
url: None,
payload: Some(serde_json::json!({
"mode": "disabled",
"reason": "DETMIR_DLP_ENABLED=false",
"checks_skipped": ["DLP health command", "DLP buckets"]
})),
error: None,
});
} else if !args.disable_dlp_health_check {
if !args.disable_dlp_health_check {
checks.push(dlp_health_check(args));
}
checks
@@ -525,7 +500,6 @@ fn run_shell_command_timeout(command: &str, timeout: Duration) -> Result<Command
let mut child = Command::new("/bin/sh")
.arg("-lc")
.arg(command)
.process_group(0)
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
@@ -536,7 +510,7 @@ fn run_shell_command_timeout(command: &str, timeout: Duration) -> Result<Command
return read_command_output(child, status.code(), false);
}
if started.elapsed() >= timeout {
terminate_process_group(child.id());
let _ = child.kill();
let _ = child.wait();
return read_command_output(child, None, true);
}
@@ -544,17 +518,6 @@ fn run_shell_command_timeout(command: &str, timeout: Duration) -> Result<Command
}
}
fn terminate_process_group(child_pid: u32) {
let process_group = format!("-{child_pid}");
let _ = Command::new("/bin/kill")
.args(["-TERM", "--", &process_group])
.status();
std::thread::sleep(Duration::from_secs(2));
let _ = Command::new("/bin/kill")
.args(["-KILL", "--", &process_group])
.status();
}
fn read_command_output(
mut child: std::process::Child,
code: Option<i32>,
@@ -833,7 +796,7 @@ fn bucket_health(args: &Cli) -> Result<Vec<BucketCheck>> {
let interactive_required = interactive_required(&client, &args.hostname, now);
let mut out = Vec::new();
for spec in bucket_specs(&args.hostname, args.dlp_enabled) {
for spec in bucket_specs(&args.hostname) {
if matches!(spec.mode, BucketMode::EventDriven) {
out.push(BucketCheck {
label: spec.label.to_string(),
@@ -1030,19 +993,6 @@ fn render_text(report: &CheckReport) -> String {
fn main() -> Result<()> {
let mut args = Cli::parse();
args.service_timeout_seconds = env_u64(
"DETMIR_SERVICE_TIMEOUT_SECONDS",
args.service_timeout_seconds,
);
args.bucket_timeout_seconds =
env_u64("DETMIR_BUCKET_TIMEOUT_SECONDS", args.bucket_timeout_seconds);
args.tcp_timeout_seconds = env_f64("DETMIR_TCP_TIMEOUT_SECONDS", args.tcp_timeout_seconds);
args.dlp_timeout_seconds = env_u64("DETMIR_DLP_TIMEOUT_SECONDS", args.dlp_timeout_seconds);
args.overall_timeout_seconds = env_u64(
"DETMIR_CHECK_OVERALL_TIMEOUT_SECONDS",
args.overall_timeout_seconds,
);
start_overall_timeout_watchdog(args.overall_timeout_seconds);
args.aw_api = env_or_default("DETMIR_AW_API", &args.aw_api);
args.worktime_url = env_or_default("DETMIR_WORKTIME_URL", &args.worktime_url);
args.one_c_url = env_or_default("DETMIR_ONE_C_URL", &args.one_c_url);
@@ -1061,12 +1011,6 @@ fn main() -> Result<()> {
if env_flag_enabled("DETMIR_DISABLE_DLP_HEALTH_CHECK") {
args.disable_dlp_health_check = true;
}
if let Some(value) = std::env::var("DETMIR_DLP_ENABLED")
.ok()
.filter(|value| !value.is_empty())
{
args.dlp_enabled = parse_env_flag(&value);
}
if env_flag_enabled("DETMIR_DISABLE_PORTAL_CHECK") {
args.disable_portal_check = true;
}
@@ -1084,31 +1028,6 @@ fn main() -> Result<()> {
});
}
fn env_u64(name: &str, fallback: u64) -> u64 {
std::env::var(name)
.ok()
.and_then(|value| value.parse().ok())
.unwrap_or(fallback)
}
fn env_f64(name: &str, fallback: f64) -> f64 {
std::env::var(name)
.ok()
.and_then(|value| value.parse().ok())
.unwrap_or(fallback)
}
fn start_overall_timeout_watchdog(seconds: u64) {
if seconds == 0 {
return;
}
std::thread::spawn(move || {
std::thread::sleep(Duration::from_secs(seconds));
eprintln!("detmir-check timed out after {seconds} seconds");
std::process::exit(124);
});
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1176,28 +1095,6 @@ mod tests {
assert!(!parse_env_flag("false"));
}
#[test]
fn dlp_disabled_removes_dlp_bucket_specs() {
let enabled = bucket_specs("HOST-EXAMPLE", true);
assert!(
enabled
.iter()
.any(|spec| spec.bucket.starts_with("aw-dlp-"))
);
let disabled = bucket_specs("HOST-EXAMPLE", false);
assert!(
disabled
.iter()
.all(|spec| !spec.bucket.starts_with("aw-dlp-"))
);
assert!(
disabled
.iter()
.any(|spec| spec.bucket.starts_with("aw-worktime-sessions_"))
);
}
#[test]
fn clickhouse_database_identifier_rejects_injection() {
assert_eq!(
-1
View File
@@ -9,4 +9,3 @@ publish.workspace = true
[dependencies]
anyhow.workspace = true
clap.workspace = true
serde_json.workspace = true
+3 -87
View File
@@ -1,6 +1,5 @@
use std::io::{self, Write};
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
use std::process::Command;
use anyhow::{Context, Result};
use clap::Parser;
@@ -20,14 +19,8 @@ struct Cli {
#[arg(long, default_value_t = 10)]
connect_timeout_seconds: u64,
#[arg(long, default_value_t = 90)]
timeout_seconds: u64,
#[arg(long, default_value = DEFAULT_REMOTE_COMMAND)]
remote_command: String,
#[arg(long, default_value_t = true)]
enabled: bool,
}
impl Cli {
@@ -38,8 +31,6 @@ impl Cli {
);
self.remote_command = env_first(&["DETMIR_DLP_REMOTE_COMMAND"], &self.remote_command);
self.ssh_bin = env_first(&["DETMIR_SSH_BIN"], &self.ssh_bin);
self.timeout_seconds = env_u64("DETMIR_DLP_TIMEOUT_SECONDS", self.timeout_seconds);
self.enabled = env_bool("DETMIR_DLP_ENABLED", self.enabled);
self
}
}
@@ -51,25 +42,6 @@ fn env_first(names: &[&str], fallback: &str) -> String {
.unwrap_or_else(|| fallback.to_string())
}
fn env_u64(name: &str, fallback: u64) -> u64 {
std::env::var(name)
.ok()
.and_then(|value| value.parse().ok())
.unwrap_or(fallback)
}
fn env_bool(name: &str, fallback: bool) -> bool {
std::env::var(name)
.ok()
.map(|value| {
matches!(
value.trim().to_ascii_lowercase().as_str(),
"1" | "true" | "yes" | "on"
)
})
.unwrap_or(fallback)
}
fn ssh_args(cli: &Cli) -> Vec<String> {
vec![
"-o".to_string(),
@@ -84,76 +56,22 @@ fn ssh_args(cli: &Cli) -> Vec<String> {
}
fn run(cli: Cli) -> Result<i32> {
if !cli.enabled {
println!(
"{}",
serde_json::to_string_pretty(&serde_json::json!({
"ok": true,
"counts": {"ok": 1, "warn": 0, "fail": 0},
"results": [{
"name": "dlp:mode",
"status": "ok",
"summary": "DLP health check disabled by DETMIR_DLP_ENABLED=false",
"details": {
"mode": "disabled",
"load_reduction": ["aw-dlp health ssh probe skipped"]
}
}]
}))?
);
return Ok(0);
}
let args = ssh_args(&cli);
let mut child = Command::new(&cli.ssh_bin)
let output = Command::new(&cli.ssh_bin)
.args(&args)
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.output()
.with_context(|| format!("failed to execute {}", cli.ssh_bin))?;
let started = Instant::now();
let mut timed_out = false;
loop {
if child.try_wait()?.is_some() {
break;
}
if started.elapsed() >= Duration::from_secs(cli.timeout_seconds) {
timed_out = true;
terminate_child(&mut child);
break;
}
std::thread::sleep(Duration::from_millis(100));
}
let output = child
.wait_with_output()
.context("failed to collect SSH output")?;
io::stdout()
.write_all(&output.stdout)
.context("failed to write DLP stdout")?;
io::stderr()
.write_all(&output.stderr)
.context("failed to write DLP stderr")?;
if timed_out {
writeln!(
io::stderr(),
"detmir-dlp timed out after {} seconds",
cli.timeout_seconds
)
.context("failed to write timeout message")?;
return Ok(124);
}
Ok(output.status.code().unwrap_or(1))
}
fn terminate_child(child: &mut std::process::Child) {
let _ = child.kill();
std::thread::sleep(Duration::from_secs(2));
let _ = child.kill();
}
fn main() -> Result<()> {
let cli = Cli::parse().apply_env();
let code = run(cli)?;
@@ -170,9 +88,7 @@ mod tests {
ssh_bin: "ssh".to_string(),
ssh_target: DEFAULT_SSH_TARGET.to_string(),
connect_timeout_seconds: 10,
timeout_seconds: 90,
remote_command: DEFAULT_REMOTE_COMMAND.to_string(),
enabled: true,
};
assert_eq!(
ssh_args(&cli),
@@ -97,6 +97,17 @@ body.security-mode .demo-button.is-active {
color: var(--link);
}
.compact-actions {
display: flex;
flex-wrap: wrap;
gap: 6px;
min-width: 220px;
}
.security-findings-table td {
vertical-align: top;
}
.shell {
max-width: none;
min-width: 0;
+305 -9
View File
@@ -9,6 +9,7 @@ const state = {
cases: null,
kpiExplain: null,
pendingScrollSelector: null,
refreshSeq: 0,
load: {
status: "LOADING",
stage: "Инициализация портала",
@@ -81,7 +82,7 @@ function apiBase() {
function apiRole() {
if (state.tab === "employees" || state.tab === "departments") return "manager";
if (state.tab === "owner" || state.tab === "perimeter") return "security";
if (state.tab === "owner" || state.tab === "perimeter" || state.tab === "securityFindings") return "security";
if (state.tab === "incidents") return "forensics";
if (state.tab === "settings") return "admin";
const mode = currentViewMode();
@@ -104,6 +105,22 @@ async function loadJson(path) {
return response.json();
}
async function loadJsonWithTimeout(path, timeoutMs) {
const controller = new AbortController();
const timer = window.setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch(`${apiBase()}${path}`, {
cache: "no-store",
headers: roleHeaders(),
signal: controller.signal
});
if (!response.ok) throw new Error(`${path}: HTTP ${response.status}`);
return response.json();
} finally {
window.clearTimeout(timer);
}
}
async function postJson(path, payload) {
const response = await fetch(`${apiBase()}${path}`, {
method: "POST",
@@ -114,6 +131,19 @@ async function postJson(path, payload) {
return response.json();
}
function fallbackSummary(error) {
return {
operator_ok: false,
severity: "STALE",
blocks: {
collection: {
status: "STALE",
text: `Портал прогревает первичный срез; быстрый summary временно недоступен: ${error?.message || "timeout"}`
}
}
};
}
function statusClass(status) {
const s = String(status || "UNKNOWN").toLowerCase();
if (s === "ok" || s === "ready" || s === "true" || s === "normal" || s === "low" || s === "false_positive" || s === "resolved") return "status-ok";
@@ -480,6 +510,9 @@ function hasTabData(tab, payload) {
if (tab === "owner" || tab === "perimeter") {
return Boolean(payload.data && Object.keys(payload.data).length > 0);
}
if (tab === "securityFindings") {
return Boolean(payload.data && Object.keys(payload.data).length > 0);
}
if (tab === "incidents") {
return (Array.isArray(payload.data?.incidents) && payload.data.incidents.length > 0)
|| (Array.isArray(payload.data?.reports?.risk_incident_candidates) && payload.data.reports.risk_incident_candidates.length > 0)
@@ -1370,6 +1403,7 @@ function renderSecurityView(data, report, extras = {}) {
return `
${renderActionCenter(report?.recommended_actions, { title: "Рекомендуемые действия ИБ", security: true })}
${renderSecurityEventsSummary(report?.security_events_summary)}
${renderSecurityFindingInbox(extras.securityFindings)}
${renderRiskIncidentCandidates(report?.risk_incident_candidates)}
${renderSecurityCorrelation(report?.security_correlation)}
${renderCases(cases)}
@@ -1382,10 +1416,84 @@ function renderSecurityView(data, report, extras = {}) {
`;
}
function renderSecurityFindingInbox(inbox) {
if (!inbox) {
return `<section class="card security-findings-card"><h3>Подозрительные станции</h3><p class="muted">Очередь подозрительных станций загружается.</p></section>`;
}
const disabled = inbox.backend === "disabled" || inbox.status === "disabled";
const fallback = Boolean(inbox.fallback_used);
const status = disabled ? "UNKNOWN" : fallback ? "WARN" : Number(inbox.critical_count || 0) > 0 ? "FAIL" : Number(inbox.open_count || 0) > 0 ? "WARN" : "OK";
const items = Array.isArray(inbox.items) ? inbox.items : [];
const rows = items.map(item => `
<tr>
<td>
<strong>${ui(item.host || "-")}</strong>
<div class="muted small">${ui(item.ip || "-")} · ${ui(item.user || "-")} · ${ui(item.department || "-")}</div>
</td>
<td><span class="badge ${statusClass(item.severity)}">${ui(item.severity || "-")}</span></td>
<td>
<strong>${ui(item.state || "-")}</strong>
<div class="muted small">${ui(item.workflow_status || "new")} · ${ui(item.last_workflow_event || "created")}</div>
</td>
<td>
<strong>${ui(item.source || "-")}</strong>
<div class="muted small">${ui(item.rule_id || "-")}</div>
</td>
<td>${ui(item.summary || item.rule_title || "-")}</td>
<td>
<div class="actions compact-actions">
<button class="small-button" data-security-finding-action="decide" data-security-finding-id="${escapeHtml(item.finding_id)}">decide</button>
<button class="small-button" data-security-finding-action="plan" data-security-finding-id="${escapeHtml(item.finding_id)}">plan</button>
<button class="small-button" data-security-finding-action="approve" data-security-finding-id="${escapeHtml(item.finding_id)}">approve</button>
<button class="small-button" data-security-finding-action="apply" data-security-finding-id="${escapeHtml(item.finding_id)}">apply</button>
<button class="small-button" data-security-finding-action="rollback" data-security-finding-id="${escapeHtml(item.finding_id)}">rollback</button>
</div>
</td>
</tr>
`).join("");
return `
<section class="card security-findings-card">
<div class="section-head">
<div>
<h3 ${tooltip("Очередь подозрительных рабочих станций из Hayabusa/Sigma/Velociraptor/AWatch. Кнопки фиксируют workflow-события, но не выполняют firewall apply.")}>Подозрительные станции</h3>
<p class="muted">Security Finding Inbox: triage -> decide -> plan -> approve -> apply. Реальное применение выполняется отдельным executor.</p>
</div>
<span class="badge ${statusClass(status)}">${ui(status)}</span>
</div>
<div class="quality-grid">
<div><span class="muted">Открыто</span><strong>${ui(inbox.open_count ?? 0)}</strong></div>
<div><span class="muted">Critical</span><strong>${ui(inbox.critical_count ?? 0)}</strong></div>
<div><span class="muted">High</span><strong>${ui(inbox.high_count ?? 0)}</strong></div>
<div><span class="muted">Contained</span><strong>${ui(inbox.contained_count ?? 0)}</strong></div>
</div>
${fallback ? `<div class="quality-warning">Security Finding Inbox временно недоступен: ${ui(inbox.error || "ошибка источника")}</div>` : ""}
${disabled ? `<p class="muted small">Security Finding Inbox отключен: включите SECURITY_EVENTS_BACKEND=clickhouse и примените схему ClickHouse.</p>` : ""}
${items.length === 0 ? `<p class="muted">Подозрительных станций в очереди нет.</p>` : `
<div class="table-scroll">
<table class="data-table security-findings-table">
<thead>
<tr>
<th>Станция</th>
<th>Риск</th>
<th>Статус</th>
<th>Источник</th>
<th>Описание</th>
<th>Workflow</th>
</tr>
</thead>
<tbody>${rows}</tbody>
</table>
</div>
`}
</section>
`;
}
function renderManagerView(report) {
return `
${renderExecutiveDashboard(report)}
${renderKpiExplain(report?.workforce_kpi_explain)}
${renderWorkforceOperations(report?.workforce_operations || report?.workforce?.operations)}
${renderDepartmentRanking(report)}
${renderDepartmentHeatMap(report)}
${renderOverviewAnalytics(report)}
@@ -1708,6 +1816,134 @@ function renderSimpleItems(items, emptyText) {
`).join("")}</div>`;
}
function renderWorkforceOperations(ops) {
if (!ops || typeof ops !== "object") return "";
const summary = ops.summary || {};
const load = summary.load || {};
const idle = summary.idle || {};
const discipline = summary.discipline || {};
const confidence = summary.confidence || {};
const rows = Array.isArray(ops.rows) ? ops.rows.slice(0, 18) : [];
const model = ops.model || {};
const guardrail = summary.guardrail || "Low confidence строки требуют проверки источников до персонального вывода.";
return `
<section class="dashboard-band workforce-ops-band">
<div class="band-head">
<div>
<h3>Операционная загрузка</h3>
<span class="muted">загрузка, простои, перегруз, дисциплина процесса и достоверность</span>
</div>
<span class="badge ${statusClass(summary.status || ops.status)}">${ui(summary.status || ops.status || "UNKNOWN")}</span>
</div>
<div class="quality-grid">
<div><span class="muted">Требуют разбора</span><strong>${ui(summary.action_required_users ?? 0)}</strong></div>
<div><span class="muted">Недогруз / ниже цели</span><strong>${ui(load.underloaded_users ?? 0)}</strong></div>
<div><span class="muted">Перегруз</span><strong>${ui(load.overloaded_users ?? 0)}</strong></div>
<div><span class="muted">Простой</span><strong>${ui(idle.idle_users ?? 0)}</strong></div>
<div><span class="muted">Дисциплина процесса</span><strong>${ui(discipline.review_users ?? 0)}</strong></div>
<div><span class="muted">Low confidence</span><strong>${ui(confidence.low_users ?? 0)}</strong></div>
</div>
<div class="table-scroll">
<table class="data-table">
<thead>
<tr>
<th>Сотрудник</th>
<th>Ответственный</th>
<th>Активно</th>
<th>Простой</th>
<th>Coverage</th>
<th>Загрузка</th>
<th>Простой</th>
<th>Дисциплина</th>
<th>Достоверность</th>
<th>Действие</th>
</tr>
</thead>
<tbody>
${rows.length ? rows.map(row => `
<tr>
<td><strong>${ui(row.user || "-")}</strong><small>${ui(row.department || "-")}</small></td>
<td>${ui(row.manager_owner || "-")}</td>
<td>${ui(row.workday_active_hhmm || "00:00")}</td>
<td>${ui(row.workday_idle_hhmm || "00:00")}</td>
<td>${ui(Math.round(Number(row.coverage_pct || 0)))}%</td>
<td><span class="badge ${statusClass(workforceOpsSeverity("load", row.load_status))}">${ui(workforceOpsLabel("load", row.load_status))}</span></td>
<td><span class="badge ${statusClass(workforceOpsSeverity("idle", row.idle_status))}">${ui(workforceOpsLabel("idle", row.idle_status))}</span></td>
<td><span class="badge ${statusClass(workforceOpsSeverity("discipline", row.discipline_status))}">${ui(workforceOpsLabel("discipline", row.discipline_status))}</span></td>
<td><span class="badge ${statusClass(workforceOpsSeverity("confidence", row.data_confidence))}">${ui(workforceOpsLabel("confidence", row.data_confidence))}</span></td>
<td>${ui(row.recommended_action || row.operations_recommended_action || row.operations?.recommended_action || "-")}</td>
</tr>
`).join("") : `
<tr><td colspan="10"><strong>Нет данных</strong><small>Управленческий срез Worktime пока не сформирован.</small></td></tr>
`}
</tbody>
</table>
</div>
<p class="muted small">${ui(guardrail)} · модель: <code>${escapeHtml(model.version || "workforce-operations-v1")}</code> / <code>${escapeHtml(model.type || "rule_based")}</code>.</p>
</section>
`;
}
function workforceOpsSeverity(kind, value) {
const status = String(value || "").toLowerCase();
if (kind === "load") {
if (status === "normal") return "OK";
if (status === "overloaded") return "HIGH";
if (["underloaded", "below_target", "no_activity"].includes(status)) return "WARN";
if (["no_data", "insufficient_data"].includes(status)) return "MISSING";
}
if (kind === "idle") {
if (status === "no_significant_idle") return "OK";
if (["idle_detected", "full_workday_idle_or_absent"].includes(status)) return "WARN";
if (status === "unknown") return "MISSING";
}
if (kind === "discipline") {
if (status === "ok") return "OK";
if (status) return "WARN";
}
if (kind === "confidence") {
if (status === "high") return "OK";
if (status === "medium") return "WARN";
if (status === "low") return "MISSING";
}
return "UNKNOWN";
}
function workforceOpsLabel(kind, value) {
const status = String(value || "").toLowerCase();
const labels = {
load: {
insufficient_data: "нет окна",
no_data: "нет данных",
no_activity: "нет активности",
underloaded: "недогруз",
below_target: "ниже цели",
normal: "норма",
overloaded: "перегруз",
},
idle: {
not_applicable: "не применимо",
unknown: "нет данных",
full_workday_idle_or_absent: "пустой день",
idle_detected: "простой",
no_significant_idle: "без простоя",
},
discipline: {
ok: "процесс в норме",
off_hours: "вне графика",
late_start: "поздний старт",
early_finish: "раннее завершение",
multiple_flags: "несколько отклонений",
},
confidence: {
high: "high",
medium: "medium",
low: "low",
},
};
return labels[kind]?.[status] || value || "unknown";
}
function workforceIndexText(usersCount, activeSeconds) {
const users = Number(usersCount || 0);
const seconds = Number(activeSeconds || 0);
@@ -3037,6 +3273,7 @@ function renderReports(data) {
<h3 class="section-title">Ключевые показатели</h3>
${renderKpiCards(data.kpis)}
${renderKpiExplain(data.workforce_kpi_explain)}
${renderWorkforceOperations(data.workforce_operations || data.workforce?.operations)}
${renderAgentQuality(data.agent_quality, data.agent_quality_explain)}
${renderAgentQualityHistory(data.agent_quality_history, data.agent_quality_history_summary)}
${renderAgentQualityNodes(data.agent_quality_nodes, data.agent_quality_nodes_summary)}
@@ -3107,28 +3344,41 @@ async function refresh(options = {}) {
const content = document.getElementById("content");
const background = Boolean(options.background);
const stage = options.stage || "Получение данных";
const refreshSeq = ++state.refreshSeq;
const isCurrentRefresh = () => refreshSeq === state.refreshSeq;
const progress = (status, label, value) => {
if (!background) setLoadStatus(status, label, value);
if (!background && isCurrentRefresh()) setLoadStatus(status, label, value);
};
try {
progress("LOADING", stage, 8);
if (!background && content) content.innerHTML = renderLoadingContent(stage);
if (!background && content && isCurrentRefresh()) {
content.innerHTML = renderLoadingContent(stage);
}
if (!state.links) {
progress("LOADING", "Получение данных", 18);
state.links = await loadJson("/links");
if (!isCurrentRefresh()) return;
}
progress("LOADING", "Расчёт показателей", 34);
const summary = await loadJson("/summary");
const summary = await loadJsonWithTimeout("/summary", 5000).catch(fallbackSummary);
if (!isCurrentRefresh()) return;
progress("LOADING", "Расчёт показателей", 46);
state.readiness = {
bundle: await loadJson("/readiness/bundle").catch(error => ({ ok: false, error: error.message })),
bundle: await loadJsonWithTimeout("/readiness/bundle", 3000).catch(error => ({ ok: false, error: error.message })),
verify: state.readiness?.verify || null
};
if (!isCurrentRefresh()) return;
renderSummary(summary, state.readiness);
progress("LOADING", "Формирование главного вывода", 68);
const tabResult = await loadCurrentTab();
if (!isCurrentRefresh()) return;
progress("LOADING", "Подготовка разделов", 88);
if (!hasTabData(state.tab, tabResult)) {
if (tabResult.html && tabResult.html.includes("data-loading-state=\"STALE\"")) {
setLoadStatus("STALE", "Первичный срез прогревается", 100, { error: "cache/prewarm in progress" });
if (content) content.innerHTML = tabResult.html;
return;
}
setLoadStatus("EMPTY", "Данные отсутствуют", 100);
if (content) {
content.innerHTML = `${renderEmptyState("Источники ответили, но полезные записи для текущего раздела пока не найдены.")}${tabResult.html || ""}`;
@@ -3139,20 +3389,29 @@ async function refresh(options = {}) {
setLoadStatus("READY", "Данные готовы", 100);
consumePendingScroll();
} catch (error) {
if (!isCurrentRefresh()) return;
showError(error);
}
}
async function loadCurrentTab() {
if (state.tab === "operator") {
const data = await loadJson("/operator");
const data = await loadJsonWithTimeout("/operator", 7000).catch(error => null);
if (!data) {
return {
data: {},
html: staleBanner("Портал прогревает первичный срез. Быстрые health/readiness доступны, тяжелый операционный срез будет подставлен после cache/prewarm.")
};
}
state.operatorData = data;
state.reports = await loadJson("/reports").catch(() => state.reports);
state.reports = await loadJsonWithTimeout("/reports", 3000).catch(() => state.reports);
let securityFindings = null;
if (currentViewMode() === "security" || currentViewMode() === "forensics") {
state.cases = await loadJson("/cases").catch(error => ({ ok: false, error: error.message, cases: [] }));
securityFindings = await loadJson("/security/findings").catch(error => ({ status: "fallback", fallback_used: true, error: error.message, items: [] }));
}
updateFilters(state.reports);
return { data, report: state.reports, cases: state.cases, html: renderOperator(data, state.reports, { cases: state.cases }) };
return { data, report: state.reports, cases: state.cases, securityFindings, html: renderOperator(data, state.reports, { cases: state.cases, securityFindings }) };
}
if (state.tab === "manager") {
const data = await loadJson("/manager");
@@ -3178,6 +3437,19 @@ async function loadCurrentTab() {
const data = await loadJson("/owner");
return { data, html: renderOwner(data) };
}
if (state.tab === "securityFindings") {
const data = await loadJson("/security/findings");
return { data, html: `
<div class="page-head">
<div>
<h2 class="section-title">Подозрительные станции</h2>
<p class="muted">Очередь triage для Hayabusa/Sigma/Velociraptor/AWatch findings. Реальное применение containment выполняется отдельным executor.</p>
</div>
<span class="badge ${statusClass(data.status)}">${ui(data.status || "unknown")}</span>
</div>
${renderSecurityFindingInbox(data)}
` };
}
if (state.tab === "incidents") {
const data = await loadJson("/incidents");
const evidence = await loadJson("/dlp/evidence").catch(error => ({ ok: false, error: error.message, items: [] }));
@@ -3213,6 +3485,7 @@ function tabLoadingStage(tab) {
employees: "Получение данных сотрудников",
departments: "Расчёт показателей подразделений",
owner: "Формирование главного вывода по рискам",
securityFindings: "Загрузка очереди подозрительных станций",
incidents: "Подготовка разделов расследований",
perimeter: "Подготовка разделов сетевого периметра",
reports: "Подготовка разделов отчета",
@@ -3230,7 +3503,7 @@ function setTab(tab) {
}
function applySecurityMode(tab) {
document.body.classList.toggle("security-mode", tab === "owner" || tab === "incidents" || tab === "perimeter");
document.body.classList.toggle("security-mode", tab === "owner" || tab === "incidents" || tab === "perimeter" || tab === "securityFindings");
}
function consumePendingScroll() {
@@ -3311,6 +3584,12 @@ document.addEventListener("click", event => {
incidentAction(button).catch(showError);
});
document.addEventListener("click", event => {
const button = event.target.closest("[data-security-finding-action]");
if (!button) return;
securityFindingWorkflowAction(button).catch(showError);
});
document.addEventListener("click", event => {
const button = event.target.closest("[data-review-status]");
if (!button) return;
@@ -3425,6 +3704,23 @@ async function incidentAction(button) {
await refresh({ stage: "Обновление статуса инцидента" });
}
async function securityFindingWorkflowAction(button) {
const findingId = button.dataset.securityFindingId;
const action = button.dataset.securityFindingAction;
const promptText = action === "apply"
? "Комментарий к apply request. Реальное применение firewall отсюда не выполняется."
: `Комментарий к действию ${action}`;
const comment = window.prompt(promptText, "");
if (comment === null) return;
button.disabled = true;
await postJson("/security/findings/workflow", {
finding_id: findingId,
action,
comment,
});
await refresh({ stage: "Обновление очереди подозрительных станций" });
}
async function candidateReviewAction(button) {
const candidateId = button.dataset.candidateId;
const status = button.dataset.reviewStatus;
@@ -18,6 +18,7 @@
<button class="tab" data-tab="employees">Сотрудники</button>
<button class="tab" data-tab="departments">Подразделения</button>
<button class="tab" data-tab="owner">Риски</button>
<button class="tab" data-tab="securityFindings">Подозрительные станции</button>
<button class="tab" data-tab="incidents">Расследования</button>
<button class="tab" data-tab="perimeter">Сетевой периметр</button>
<button class="tab" data-tab="reports">Отчеты</button>
+28 -210
View File
@@ -62,21 +62,6 @@ struct Cli {
#[arg(long)]
json: bool,
#[arg(long, default_value_t = 120)]
overall_timeout_seconds: u64,
#[arg(long, default_value = "full")]
profile: String,
#[arg(long, default_value_t = true)]
enabled: bool,
#[arg(long, default_value = "operator_disabled")]
disabled_reason: String,
#[arg(long, default_value = "")]
disabled_since: String,
}
impl Cli {
@@ -144,28 +129,6 @@ impl Cli {
if !cli_arg_present("--profiles") {
self.profiles = env_string("AW_DLP_COMPLIANCE_PROFILES").unwrap_or(self.profiles);
}
if !cli_arg_present("--overall-timeout-seconds") {
self.overall_timeout_seconds = env_u64(
"AW_DLP_HEALTH_OVERALL_TIMEOUT_SECONDS",
self.overall_timeout_seconds,
);
}
if !cli_arg_present("--profile") {
self.profile = env_string("AW_DLP_PROFILE")
.or_else(|| env_string("DETMIR_PORTAL_DLP_PROFILE"))
.unwrap_or(self.profile);
}
if !cli_arg_present("--enabled") {
self.enabled = env_bool_default("AW_DLP_ENABLED", self.enabled);
}
if !cli_arg_present("--disabled-reason") {
self.disabled_reason =
env_string("AW_DLP_DISABLED_REASON").unwrap_or(self.disabled_reason);
}
if !cli_arg_present("--disabled-since") {
self.disabled_since =
env_string("AW_DLP_DISABLED_SINCE").unwrap_or(self.disabled_since);
}
self
}
}
@@ -1307,66 +1270,8 @@ fn check_compliance_reports(
}
}
fn normalized_profile(profile: &str) -> String {
match profile.trim().to_ascii_lowercase().as_str() {
"disabled" | "off" => "core_only".to_string(),
"core-only" | "core_only" => "core_only".to_string(),
"light" | "lite" => "light".to_string(),
"on-demand" | "on_demand" => "on_demand".to_string(),
"enabled" | "on" | "full" => "full".to_string(),
"" => "full".to_string(),
other => other.to_string(),
}
}
fn profile_is_disabled(profile: &str) -> bool {
matches!(normalized_profile(profile).as_str(), "core_only")
}
fn profile_checks_heavy_services(profile: &str) -> bool {
matches!(normalized_profile(profile).as_str(), "full" | "on_demand")
}
fn build_report(cli: &Cli, client: &Client) -> HealthReport {
let mut report = HealthReport::default();
let profile = normalized_profile(&cli.profile);
if !cli.enabled || profile_is_disabled(&profile) {
report.add(
"dlp:mode",
"ok",
"DLP runtime disabled by production profile",
json!({
"mode": "disabled",
"profile": profile,
"reason": &cli.disabled_reason,
"disabled_since": empty_string_as_null(&cli.disabled_since),
"checks_skipped": [
"policy API",
"case API",
"DLP systemd units",
"DLP ActivityWatch buckets",
"DLP compliance reports"
],
"load_reduction": [
"no DLP bucket freshness reads",
"no DLP case/policy HTTP checks",
"no DLP compliance filesystem scan"
]
}),
);
return report;
}
report.add(
"dlp:mode",
"ok",
format!("DLP runtime profile {profile}"),
json!({
"mode": if profile == "light" { "light" } else { "enabled" },
"profile": profile,
"heavy_services_checked": profile_checks_heavy_services(&cli.profile)
}),
);
let aw_api_base = format!("{}/api/0", cli.aw_server.trim_end_matches('/'));
let counter_state_path = cli.state_dir.join("dlp-health-check-counters.json");
let mut counter_state = load_counter_state(&counter_state_path);
@@ -1377,70 +1282,38 @@ fn build_report(cli: &Cli, client: &Client) -> HealthReport {
"http:aw",
&format!("{aw_api_base}/info"),
);
if profile_checks_heavy_services(&cli.profile) {
check_http_endpoint(
&mut report,
client,
"http:policy",
&format!("{}/healthz", cli.policy_server.trim_end_matches('/')),
);
check_http_endpoint(
&mut report,
client,
"http:cases",
&format!("{}/health", cli.case_server.trim_end_matches('/')),
);
} else {
report.add(
"http:heavy-dlp",
"ok",
"heavy DLP policy/case HTTP checks skipped for lightweight profile",
json!({
"profile": profile,
"skipped": ["policy API", "case API"]
}),
);
}
check_http_endpoint(
&mut report,
client,
"http:policy",
&format!("{}/healthz", cli.policy_server.trim_end_matches('/')),
);
check_http_endpoint(
&mut report,
client,
"http:cases",
&format!("{}/health", cli.case_server.trim_end_matches('/')),
);
for unit in ["activitywatch-server", "aw-worktime-api.service"] {
for unit in [
"activitywatch-server",
"aw-dlp-policy-engine.service",
"aw-dlp-case-management.service",
"aw-worktime-api.service",
] {
check_systemd_unit(&mut report, unit, "service");
}
if profile_checks_heavy_services(&cli.profile) {
for unit in [
"aw-dlp-policy-engine.service",
"aw-dlp-case-management.service",
] {
check_systemd_unit(&mut report, unit, "service");
}
for unit in [
"aw-dlp-report-scheduler.timer",
"aw-dlp-syslog-forwarder.timer",
"aw-dlp-webhook-sender.timer",
"aw-dlp-cef-exporter.timer",
"activitywatch-dlp-aggregator.timer",
"aw-dlp-ioc-refresh.timer",
] {
check_systemd_unit(&mut report, unit, "timer");
}
} else {
report.add(
"systemd:heavy-dlp",
"ok",
"heavy DLP systemd checks skipped for lightweight profile",
json!({
"profile": profile,
"skipped": [
"aw-dlp-policy-engine.service",
"aw-dlp-case-management.service",
"aw-dlp-report-scheduler.timer",
"aw-dlp-syslog-forwarder.timer",
"aw-dlp-webhook-sender.timer",
"aw-dlp-cef-exporter.timer"
]
}),
);
for unit in [
"aw-dlp-report-scheduler.timer",
"aw-dlp-syslog-forwarder.timer",
"aw-dlp-webhook-sender.timer",
"aw-dlp-cef-exporter.timer",
"activitywatch-dlp-aggregator.timer",
"aw-dlp-ioc-refresh.timer",
"aw-worktime-ui-bridge.timer",
] {
check_systemd_unit(&mut report, unit, "timer");
}
check_systemd_unit(&mut report, "aw-worktime-ui-bridge.timer", "timer");
match http_json(client, &format!("{aw_api_base}/buckets"), 15, 2) {
Ok(Value::Object(map)) => {
@@ -1589,12 +1462,6 @@ fn env_i64(name: &str, default: i64) -> i64 {
.unwrap_or(default)
}
fn env_u64(name: &str, default: u64) -> u64 {
env_string(name)
.and_then(|value| value.parse::<u64>().ok())
.unwrap_or(default)
}
fn env_bool(name: &str) -> bool {
env_string(name)
.map(|value| {
@@ -1606,39 +1473,8 @@ fn env_bool(name: &str) -> bool {
.unwrap_or(false)
}
fn env_bool_default(name: &str, default: bool) -> bool {
env_string(name)
.map(|value| {
matches!(
value.to_ascii_lowercase().as_str(),
"1" | "true" | "yes" | "on"
)
})
.unwrap_or(default)
}
fn empty_string_as_null(value: &str) -> Value {
if value.trim().is_empty() {
Value::Null
} else {
json!(value)
}
}
fn start_overall_timeout_watchdog(seconds: u64) {
if seconds == 0 {
return;
}
std::thread::spawn(move || {
sleep(Duration::from_secs(seconds));
eprintln!("dlp-health-check timed out after {seconds} seconds");
std::process::exit(124);
});
}
fn main() -> Result<()> {
let cli = Cli::parse().apply_env();
start_overall_timeout_watchdog(cli.overall_timeout_seconds);
let client = Client::builder()
.no_proxy()
.build()
@@ -1711,22 +1547,4 @@ mod tests {
assert_eq!(payload.counts.warn, 1);
assert_eq!(payload.counts.fail, 0);
}
#[test]
fn dlp_profile_normalization_matches_runtime_control_names() {
assert_eq!(normalized_profile("disabled"), "core_only");
assert_eq!(normalized_profile("core-only"), "core_only");
assert_eq!(normalized_profile("light"), "light");
assert_eq!(normalized_profile("lite"), "light");
assert_eq!(normalized_profile("on-demand"), "on_demand");
assert_eq!(normalized_profile("enabled"), "full");
}
#[test]
fn light_profile_does_not_require_heavy_services() {
assert!(!profile_checks_heavy_services("light"));
assert!(!profile_checks_heavy_services("core_only"));
assert!(profile_checks_heavy_services("on_demand"));
assert!(profile_checks_heavy_services("full"));
}
}
+13 -514
View File
@@ -77,7 +77,6 @@ struct Config {
management_history_retention_days: i64,
true_active_evidence_window_seconds: i64,
true_active_max_event_seconds: i64,
dlp_evidence_enabled: bool,
offset: FixedOffset,
}
@@ -228,14 +227,6 @@ fn threshold_to_pct(value: f64) -> f64 {
}
}
fn overload_threshold_to_pct(value: f64) -> f64 {
if (0.0..=3.0).contains(&value) {
value * 100.0
} else {
value
}
}
fn parse_hhmm(value: &str) -> Option<NaiveTime> {
NaiveTime::parse_from_str(value.trim(), "%H:%M").ok()
}
@@ -271,7 +262,7 @@ fn load_config() -> Config {
}
if let Some(value) = policy.overload_threshold {
manager_overload_coverage_pct =
overload_threshold_to_pct(value).round().clamp(100.0, 300.0) as i64;
threshold_to_pct(value).round().clamp(1.0, 300.0) as i64;
}
if let Some(value) = policy.drop_threshold_pct {
manager_trend_delta_pct = threshold_to_pct(value).clamp(1.0, 100.0);
@@ -380,10 +371,6 @@ fn load_config() -> Config {
.max(30),
true_active_max_event_seconds: env_i64("AW_WORKTIME_TRUE_ACTIVE_MAX_EVENT_SECONDS", 600)
.max(30),
dlp_evidence_enabled: env_bool(
"AW_WORKTIME_DLP_EVIDENCE_ENABLED",
env_bool("AW_DLP_ENABLED", true),
),
offset: FixedOffset::east_opt(3 * 3600).expect("valid Moscow offset"),
}
}
@@ -1023,7 +1010,13 @@ impl App {
}
};
let mut evidence = HashMap::new();
for bucket in evidence_bucket_ids(&self.config, host) {
for bucket in [
format!("aw-file-operations_{host}"),
format!("aw-dlp-endpoint-signals_{host}"),
format!("aw-watcher-web-chrome_{host}"),
format!("aw-watcher-web-edge_{host}"),
format!("aw-detmir-web-category_{host}"),
] {
evidence.insert(
bucket.clone(),
self.fetch_bucket_events(&bucket, Some(bounds.0), Some(bounds.1)),
@@ -1163,19 +1156,6 @@ fn sanitize_bucket_for_log(bucket_id: &str) -> String {
bucket_id.to_string()
}
fn evidence_bucket_ids(config: &Config, host: &str) -> Vec<String> {
let mut buckets = vec![format!("aw-file-operations_{host}")];
if config.dlp_evidence_enabled {
buckets.push(format!("aw-dlp-endpoint-signals_{host}"));
}
buckets.extend([
format!("aw-watcher-web-chrome_{host}"),
format!("aw-watcher-web-edge_{host}"),
format!("aw-detmir-web-category_{host}"),
]);
buckets
}
fn sanitize_error_for_log(value: &str) -> String {
static IP_RE: OnceLock<Regex> = OnceLock::new();
static BUCKET_HOST_RE: OnceLock<Regex> = OnceLock::new();
@@ -1625,156 +1605,6 @@ fn interval_overlap_seconds(
(total, first, last)
}
#[derive(Clone, Copy)]
struct EmployeeOperationsInput {
expected_seconds: i64,
low_seconds: i64,
target_seconds: i64,
overload_seconds: i64,
work_secs: i64,
calendar_secs: i64,
work_first: Option<DateTime<Utc>>,
work_last: Option<DateTime<Utc>>,
is_today: bool,
late_start: DateTime<FixedOffset>,
early_finish: DateTime<FixedOffset>,
samples_count: i64,
active_samples: i64,
sessions_count: i64,
}
fn build_employee_operations_status(config: &Config, input: EmployeeOperationsInput) -> Value {
let workday_idle_seconds = (input.expected_seconds - input.work_secs).max(0);
let off_hours_seconds = (input.calendar_secs - input.work_secs).max(0);
let coverage = if input.expected_seconds > 0 {
clamp_pct(input.work_secs as f64 / input.expected_seconds as f64 * 100.0)
} else {
0.0
};
let load_status = if input.expected_seconds <= 0 {
"insufficient_data"
} else if input.sessions_count <= 0 || input.samples_count <= 0 {
"no_data"
} else if input.work_secs <= 0 {
"no_activity"
} else if input.work_secs < input.low_seconds {
"underloaded"
} else if input.work_secs >= input.overload_seconds {
"overloaded"
} else if input.work_secs < input.target_seconds {
"below_target"
} else {
"normal"
};
let idle_status = if input.expected_seconds <= 0 {
"not_applicable"
} else if input.sessions_count <= 0 || input.samples_count <= 0 {
"unknown"
} else if input.work_secs <= 0 {
"full_workday_idle_or_absent"
} else if workday_idle_seconds >= config.manager_off_hours_threshold_seconds {
"idle_detected"
} else {
"no_significant_idle"
};
let mut discipline_flags = Vec::new();
if off_hours_seconds >= config.manager_off_hours_threshold_seconds {
discipline_flags.push("off_hours");
}
if input
.work_first
.is_some_and(|dt| dt.with_timezone(&config.offset) > input.late_start)
{
discipline_flags.push("late_start");
}
if !input.is_today
&& input
.work_last
.is_some_and(|dt| dt.with_timezone(&config.offset) < input.early_finish)
{
discipline_flags.push("early_finish");
}
let discipline_status = match discipline_flags.as_slice() {
[] => "ok",
[single] => single,
_ => "multiple_flags",
};
let mut confidence_reasons = Vec::new();
if input.sessions_count <= 0 {
confidence_reasons.push("missing_session_samples");
}
if input.samples_count <= 0 {
confidence_reasons.push("missing_worktime_samples");
} else if input.samples_count < 3 {
confidence_reasons.push("few_worktime_samples");
} else {
confidence_reasons.push("worktime_samples_present");
}
if input.active_samples <= 0 {
confidence_reasons.push("missing_active_samples");
} else {
confidence_reasons.push("active_samples_present");
}
if input.expected_seconds <= 0 {
confidence_reasons.push("workday_window_not_started_or_empty");
}
let data_confidence =
if input.expected_seconds <= 0 || input.sessions_count <= 0 || input.samples_count <= 0 {
"low"
} else if input.samples_count < 3 || input.active_samples <= 0 {
"medium"
} else {
"high"
};
let recommended_action = match (data_confidence, load_status, discipline_status, idle_status) {
("low", _, _, _) => {
"Сначала проверить свежесть источников и наличие сессии; вывод по сотруднику не использовать как дисциплинарный."
}
(_, "overloaded", _, _) => {
"Проверить переработку, перераспределение задач и риск аврального процесса."
}
(_, "underloaded" | "below_target" | "no_activity", _, _) => {
"Проверить фактическую загрузку, задачи, доступ к рабочим системам и отсутствие сбоя сбора."
}
(_, _, "off_hours" | "late_start" | "early_finish" | "multiple_flags", _) => {
"Проверить согласование рабочего графика и причину отклонения от процесса."
}
(_, _, _, "idle_detected" | "full_workday_idle_or_absent") => {
"Проверить простой: отсутствие задач, ожидание внешнего процесса или техническую проблему."
}
_ => "Наблюдать; отклонений, требующих немедленного действия, не выявлено.",
};
json!({
"load_status": load_status,
"idle_status": idle_status,
"discipline_status": discipline_status,
"discipline_flags": discipline_flags,
"data_confidence": data_confidence,
"confidence_reasons": confidence_reasons,
"workday_idle_seconds": workday_idle_seconds,
"workday_idle_hhmm": hhmm(workday_idle_seconds),
"off_hours_seconds": off_hours_seconds,
"off_hours_hhmm": hhmm(off_hours_seconds),
"coverage_pct": coverage,
"evidence": {
"expected_hhmm": hhmm(input.expected_seconds),
"workday_active_hhmm": hhmm(input.work_secs),
"calendar_active_hhmm": hhmm(input.calendar_secs),
"sessions_count": input.sessions_count,
"samples_count": input.samples_count,
"active_samples": input.active_samples,
"first_workday_activity_local": input.work_first.map(|dt| dt.with_timezone(&config.offset).to_rfc3339()).unwrap_or_default(),
"last_workday_activity_local": input.work_last.map(|dt| dt.with_timezone(&config.offset).to_rfc3339()).unwrap_or_default()
},
"guardrail": if data_confidence == "low" {
"low_confidence_not_for_discipline"
} else {
"evidence_backed_manual_review_only"
},
"recommended_action": recommended_action,
})
}
impl App {
fn build_management_payload(
&self,
@@ -1804,7 +1634,6 @@ impl App {
};
let target_seconds = expected_seconds * self.config.manager_target_coverage_pct / 100;
let low_seconds = expected_seconds * self.config.manager_low_coverage_pct / 100;
let overload_seconds = expected_seconds * self.config.manager_overload_coverage_pct / 100;
let late_start =
work_start_local + TimeDelta::minutes(self.config.manager_late_start_grace_minutes);
let early_finish =
@@ -1877,37 +1706,6 @@ impl App {
.map(|dt| dt.with_timezone(&self.config.offset))
.map(|dt| dt.to_rfc3339())
.unwrap_or_default();
let samples_count = row
.get("samples_count")
.and_then(Value::as_i64)
.unwrap_or(0);
let active_samples = row
.get("active_samples")
.and_then(Value::as_i64)
.unwrap_or(0);
let sessions_count = row
.get("sessions_count")
.and_then(Value::as_i64)
.unwrap_or(0);
let operations = build_employee_operations_status(
&self.config,
EmployeeOperationsInput {
expected_seconds,
low_seconds,
target_seconds,
overload_seconds,
work_secs,
calendar_secs,
work_first,
work_last,
is_today,
late_start,
early_finish,
samples_count,
active_samples,
sessions_count,
},
);
let mut public = row.as_object().cloned().unwrap_or_default();
public.remove("_intervals");
public.insert("user".into(), json!(alias.display_name));
@@ -1921,52 +1719,8 @@ impl App {
public.insert("calendar_active_hhmm".into(), json!(hhmm(calendar_secs)));
public.insert("workday_active_seconds".into(), json!(work_secs));
public.insert("workday_active_hhmm".into(), json!(hhmm(work_secs)));
public.insert(
"workday_idle_seconds".into(),
json!((expected_seconds - work_secs).max(0)),
);
public.insert(
"workday_idle_hhmm".into(),
json!(hhmm((expected_seconds - work_secs).max(0))),
);
public.insert("coverage_pct".into(), json!(coverage));
public.insert("status".into(), json!(status));
public.insert(
"load_status".into(),
operations
.get("load_status")
.cloned()
.unwrap_or_else(|| json!("unknown")),
);
public.insert(
"idle_status".into(),
operations
.get("idle_status")
.cloned()
.unwrap_or_else(|| json!("unknown")),
);
public.insert(
"discipline_status".into(),
operations
.get("discipline_status")
.cloned()
.unwrap_or_else(|| json!("unknown")),
);
public.insert(
"data_confidence".into(),
operations
.get("data_confidence")
.cloned()
.unwrap_or_else(|| json!("low")),
);
public.insert(
"operations_recommended_action".into(),
operations
.get("recommended_action")
.cloned()
.unwrap_or_else(|| json!("Проверить первичные источники.")),
);
public.insert("operations".into(), operations.clone());
public.insert("first_activity_local".into(), json!(first_local));
public.insert("last_activity_local".into(), json!(last_local));
public.insert(
@@ -1985,10 +1739,6 @@ impl App {
"manager_owner": public.get("manager_owner").cloned().unwrap_or(json!("")),
"department": public.get("department").cloned().unwrap_or(json!("")),
"role": public.get("role").cloned().unwrap_or(json!("")),
"load_status": public.get("load_status").cloned().unwrap_or(json!("unknown")),
"idle_status": public.get("idle_status").cloned().unwrap_or(json!("unknown")),
"discipline_status": public.get("discipline_status").cloned().unwrap_or(json!("unknown")),
"data_confidence": public.get("data_confidence").cloned().unwrap_or(json!("low")),
});
let owner = public
.get("manager_owner")
@@ -2013,9 +1763,6 @@ impl App {
} else if expected_seconds > 0 && work_secs < target_seconds {
actions.push(action("target_gap_review", "medium", &owner, "24h", &format!("У сотрудника {display} активное время в рабочем окне {} ниже управленческого целевого порога {}%.", hhmm(work_secs), self.config.manager_target_coverage_pct), &format!("Уточнить причину отклонения по сотруднику {display} и подтвердить план работ."), &canonical, evidence.clone()));
}
if expected_seconds > 0 && work_secs >= overload_seconds {
actions.push(action("overload_review", "high", &owner, "24h", &format!("У сотрудника {display} активное время в рабочем окне {} выше порога перегруза {}%.", hhmm(work_secs), self.config.manager_overload_coverage_pct), &format!("Проверить переработку сотрудника {display}, распределение задач и риск аврального процесса."), &canonical, evidence.clone()));
}
if work_first.is_some_and(|dt| dt.with_timezone(&self.config.offset) > late_start) {
actions.push(action("late_start_review", "medium", &owner, "24h", &format!("У сотрудника {display} первая активность в рабочем окне зафиксирована поздно."), &format!("Проверить причину позднего старта сотрудника {display} и подтвердить, что это не проблема доступа или дисциплины."), &canonical, evidence.clone()));
}
@@ -2063,37 +1810,6 @@ impl App {
)
});
let summary = summarize_management_rows(&roster, &actions, expected_seconds);
let workforce_operations = json!({
"status": summary.pointer("/workforce_operations/status").cloned().unwrap_or_else(|| json!("LOW_CONFIDENCE")),
"summary": summary.pointer("/workforce_operations").cloned().unwrap_or_else(|| json!({})),
"rows": roster.iter().map(|row| {
json!({
"user": row.get("user").cloned().unwrap_or(json!("")),
"manager_owner": row.get("manager_owner").cloned().unwrap_or(json!("")),
"department": row.get("department").cloned().unwrap_or(json!("")),
"role": row.get("role").cloned().unwrap_or(json!("")),
"load_status": row.get("load_status").cloned().unwrap_or(json!("unknown")),
"idle_status": row.get("idle_status").cloned().unwrap_or(json!("unknown")),
"discipline_status": row.get("discipline_status").cloned().unwrap_or(json!("unknown")),
"data_confidence": row.get("data_confidence").cloned().unwrap_or(json!("low")),
"coverage_pct": row.get("coverage_pct").cloned().unwrap_or(json!(0.0)),
"workday_active_hhmm": row.get("workday_active_hhmm").cloned().unwrap_or(json!("00:00")),
"workday_idle_hhmm": row.get("workday_idle_hhmm").cloned().unwrap_or(json!("00:00")),
"recommended_action": row.get("operations_recommended_action").cloned().unwrap_or(json!("Проверить первичные источники."))
})
}).collect::<Vec<_>>(),
"model": {
"type": "rule_based",
"ml": false,
"llm": false,
"version": "workforce-operations-v1"
},
"guardrails": [
"Строки low confidence требуют проверки источников до персонального вывода",
"Отсутствие данных не считается простоем",
"Все статусы предназначены только для ручного операционного разбора"
]
});
let owner_rollups = build_rollups(&roster, &actions, "manager_owner");
let department_rollups = build_rollups(&roster, &actions, "department");
let owner_roster = owner_rollups.clone();
@@ -2149,10 +1865,8 @@ impl App {
"expected_hhmm_per_user": hhmm(expected_seconds),
"target_coverage_pct": self.config.manager_target_coverage_pct,
"low_coverage_pct": self.config.manager_low_coverage_pct,
"overload_coverage_pct": self.config.manager_overload_coverage_pct,
},
"summary": summary,
"workforce_operations": workforce_operations,
"actions": actions,
"rows": roster,
"sources": sources,
@@ -2615,7 +2329,6 @@ fn summarize_management_rows(rows: &[Value], actions: &[Value], expected_seconds
.and_then(Value::as_i64)
.unwrap_or(0)
});
let operations_summary = summarize_operations(rows);
json!({
"users_count": users_count,
"active_users": active_users,
@@ -2636,85 +2349,6 @@ fn summarize_management_rows(rows: &[Value], actions: &[Value], expected_seconds
"last_activity": rows.iter().filter_map(|r| r.get("workday_last_activity_local").and_then(Value::as_str)).filter(|s| !s.is_empty()).max().unwrap_or(""),
"top_user": top.and_then(|r| r.get("user")).and_then(Value::as_str).unwrap_or(""),
"top_user_active_hhmm": top.and_then(|r| r.get("workday_active_hhmm")).and_then(Value::as_str).unwrap_or("00:00"),
"workforce_operations": operations_summary,
})
}
fn row_str<'a>(row: &'a Value, key: &str) -> &'a str {
row.get(key).and_then(Value::as_str).unwrap_or("")
}
fn count_rows_by(rows: &[Value], key: &str, expected: &[&str]) -> i64 {
rows.iter()
.filter(|row| expected.contains(&row_str(row, key)))
.count() as i64
}
fn summarize_operations(rows: &[Value]) -> Value {
let users_count = rows.len() as i64;
let low_confidence_users = count_rows_by(rows, "data_confidence", &["low"]);
let medium_confidence_users = count_rows_by(rows, "data_confidence", &["medium"]);
let high_confidence_users = count_rows_by(rows, "data_confidence", &["high"]);
let unknown_or_no_data_users = count_rows_by(
rows,
"load_status",
&["no_data", "insufficient_data", "no_activity"],
);
let underloaded_users = count_rows_by(rows, "load_status", &["underloaded", "below_target"]);
let normal_users = count_rows_by(rows, "load_status", &["normal"]);
let overloaded_users = count_rows_by(rows, "load_status", &["overloaded"]);
let idle_users = count_rows_by(
rows,
"idle_status",
&["idle_detected", "full_workday_idle_or_absent"],
);
let discipline_review_users = rows
.iter()
.filter(|row| !matches!(row_str(row, "discipline_status"), "" | "ok"))
.count() as i64;
let action_required_users = rows
.iter()
.filter(|row| {
!matches!(row_str(row, "load_status"), "normal" | "")
|| !matches!(row_str(row, "discipline_status"), "ok" | "")
|| matches!(
row_str(row, "idle_status"),
"idle_detected" | "full_workday_idle_or_absent"
)
|| row_str(row, "data_confidence") == "low"
})
.count() as i64;
let status = if users_count == 0 || low_confidence_users == users_count {
"LOW_CONFIDENCE"
} else if overloaded_users > 0 || discipline_review_users > 0 || idle_users > 0 {
"ATTENTION"
} else if underloaded_users > 0 || unknown_or_no_data_users > 0 || medium_confidence_users > 0 {
"WATCH"
} else {
"OK"
};
json!({
"status": status,
"users_count": users_count,
"action_required_users": action_required_users,
"load": {
"unknown_or_no_data_users": unknown_or_no_data_users,
"underloaded_users": underloaded_users,
"normal_users": normal_users,
"overloaded_users": overloaded_users
},
"idle": {
"idle_users": idle_users
},
"discipline": {
"review_users": discipline_review_users
},
"confidence": {
"low_users": low_confidence_users,
"medium_users": medium_confidence_users,
"high_users": high_confidence_users
},
"guardrail": "Строки low confidence требуют проверки источников до персонального вывода"
})
}
@@ -2735,11 +2369,6 @@ fn build_rollups(rows: &[Value], actions: &[Value], field: &str) -> Vec<Value> {
"active_users",
"inactive_users",
"below_target_users",
"underloaded_users",
"overloaded_users",
"idle_users",
"discipline_review_users",
"low_confidence_users",
"workday_total_active_seconds",
"actions_count",
"critical_actions_count",
@@ -2761,24 +2390,6 @@ fn build_rollups(rows: &[Value], actions: &[Value], field: &str) -> Vec<Value> {
if row.get("status").and_then(Value::as_str) == Some("below_target") {
inc(group, "below_target_users", 1);
}
if matches!(row_str(row, "load_status"), "underloaded" | "below_target") {
inc(group, "underloaded_users", 1);
}
if row_str(row, "load_status") == "overloaded" {
inc(group, "overloaded_users", 1);
}
if matches!(
row_str(row, "idle_status"),
"idle_detected" | "full_workday_idle_or_absent"
) {
inc(group, "idle_users", 1);
}
if !matches!(row_str(row, "discipline_status"), "" | "ok") {
inc(group, "discipline_review_users", 1);
}
if row_str(row, "data_confidence") == "low" {
inc(group, "low_confidence_users", 1);
}
inc(
group,
"workday_total_active_seconds",
@@ -3739,9 +3350,9 @@ fn render_management_html(payload: &Value) -> String {
.collect()
};
let user_rows = if rows.is_empty() {
"<tr><td colspan='12'>Нет сотрудников в выборке.</td></tr>".to_string()
"<tr><td colspan='8'>Нет сотрудников в выборке.</td></tr>".to_string()
} else {
rows.iter().map(|r| format!("<tr><td>{}</td><td>{}</td><td>{}</td><td>{}</td><td class='good'>{}</td><td>{}</td><td>{}%</td><td>{}</td><td>{}</td><td>{}</td><td>{}</td><td>{}</td></tr>", esc(r["user"].as_str().unwrap_or("")), esc(r["manager_owner"].as_str().unwrap_or("")), esc(r["department"].as_str().unwrap_or("")), esc(r["status"].as_str().unwrap_or("")), esc(r["workday_active_hhmm"].as_str().unwrap_or("")), esc(r["workday_idle_hhmm"].as_str().unwrap_or("00:00")), r["coverage_pct"].as_f64().unwrap_or(0.0), esc(r["load_status"].as_str().unwrap_or("unknown")), esc(r["idle_status"].as_str().unwrap_or("unknown")), esc(r["discipline_status"].as_str().unwrap_or("unknown")), esc(r["data_confidence"].as_str().unwrap_or("low")), esc(r["operations_recommended_action"].as_str().unwrap_or("")))).collect()
rows.iter().map(|r| format!("<tr><td>{}</td><td>{}</td><td>{}</td><td>{}</td><td class='good'>{}</td><td>{}%</td><td>{}</td><td>{}</td></tr>", esc(r["user"].as_str().unwrap_or("")), esc(r["manager_owner"].as_str().unwrap_or("")), esc(r["department"].as_str().unwrap_or("")), esc(r["status"].as_str().unwrap_or("")), esc(r["workday_active_hhmm"].as_str().unwrap_or("")), r["coverage_pct"].as_f64().unwrap_or(0.0), esc(r["workday_first_activity_local"].as_str().unwrap_or("")), esc(r["workday_last_activity_local"].as_str().unwrap_or("")))).collect()
};
let source_rows = sources
.iter()
@@ -3758,7 +3369,7 @@ fn render_management_html(payload: &Value) -> String {
})
.collect::<String>();
format!(
r#"<!doctype html><html lang="ru"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>AW-rus Управленческий отчёт по работе в RDP</title><style>{}</style></head><body><main><section class="hero"><h1>AW-rus Управленческий отчёт по работе в RDP</h1><p>{} · {} · {}</p><h2>Что делать сегодня</h2><p>{}</p><nav><a href="/reports/worktime/management?format=json&host={}">JSON</a><a href="/reports/worktime/management?format=csv&host={}">CSV</a><a href="/reports/worktime/today?format=html&host={}">Классический отчёт</a><a href="/reports/worktime/management?format=html&host={}">Сбросить</a></nav></section><section><h2>Очередь действий руководителя</h2><table><tbody>{}</tbody></table></section><section><h2>Рабочая активность сотрудников</h2><p>Статусы загрузки, простоя, дисциплины процесса и достоверности. Low confidence означает: сначала проверить источники, не делать персональный вывод.</p><table><thead><tr><th>Сотрудник</th><th>Ответственный</th><th>Подразделение</th><th>Статус</th><th>Активно</th><th>Простой</th><th>Coverage</th><th>Загрузка</th><th>Простой</th><th>Дисциплина</th><th>Confidence</th><th>Действие</th></tr></thead><tbody>{}</tbody></table></section><section><h2>Тренд за период</h2><p>Тренд за {} дней</p></section><section><h2>По ответственным</h2><pre>{}</pre></section><section><h2>Ответственные и эскалация</h2><pre>{}</pre></section><section><h2>По подразделениям</h2><pre>{}</pre></section><section><h2>Свежесть источников данных</h2><table><tbody>{}</tbody></table></section><p>Фильтр: {}</p></main></body></html>"#,
r#"<!doctype html><html lang="ru"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>AW-rus Управленческий отчёт по работе в RDP</title><style>{}</style></head><body><main><section class="hero"><h1>AW-rus Управленческий отчёт по работе в RDP</h1><p>{} · {} · {}</p><h2>Что делать сегодня</h2><p>{}</p><nav><a href="/reports/worktime/management?format=json&host={}">JSON</a><a href="/reports/worktime/management?format=csv&host={}">CSV</a><a href="/reports/worktime/today?format=html&host={}">Классический отчёт</a><a href="/reports/worktime/management?format=html&host={}">Сбросить</a></nav></section><section><h2>Очередь действий руководителя</h2><table><tbody>{}</tbody></table></section><section><h2>Сотрудники</h2><table><tbody>{}</tbody></table></section><section><h2>Тренд за период</h2><p>Тренд за {} дней</p></section><section><h2>По ответственным</h2><pre>{}</pre></section><section><h2>Ответственные и эскалация</h2><pre>{}</pre></section><section><h2>По подразделениям</h2><pre>{}</pre></section><section><h2>Свежесть источников данных</h2><table><tbody>{}</tbody></table></section><p>Фильтр: {}</p></main></body></html>"#,
base_css(),
esc(payload["host"].as_str().unwrap_or("")),
esc(payload["report_date"].as_str().unwrap_or("")),
@@ -4028,36 +3639,6 @@ mod tests {
);
}
#[test]
fn dlp_evidence_bucket_is_optional_for_true_active_hot_path() {
let mut cfg = test_config();
cfg.dlp_evidence_enabled = false;
let buckets = evidence_bucket_ids(&cfg, "SHARKON2025");
assert!(
buckets
.iter()
.any(|bucket| bucket == "aw-file-operations_SHARKON2025")
);
assert!(
buckets
.iter()
.any(|bucket| bucket == "aw-watcher-web-chrome_SHARKON2025")
);
assert!(
!buckets
.iter()
.any(|bucket| bucket == "aw-dlp-endpoint-signals_SHARKON2025")
);
cfg.dlp_evidence_enabled = true;
let buckets = evidence_bucket_ids(&cfg, "SHARKON2025");
assert!(
buckets
.iter()
.any(|bucket| bucket == "aw-dlp-endpoint-signals_SHARKON2025")
);
}
#[test]
fn management_insights_detect_falling_portfolio_trend() {
let cfg = test_config();
@@ -4177,85 +3758,6 @@ mod tests {
assert_eq!(rollups[0]["actions_count"], 1);
}
#[test]
fn employee_operations_detects_overload_and_off_hours() {
let cfg = test_config();
let report_date = NaiveDate::from_ymd_opt(2026, 5, 14).unwrap();
let (work_start, work_end, expected_seconds) = workday_bounds(&cfg, report_date);
let input = EmployeeOperationsInput {
expected_seconds,
low_seconds: expected_seconds * cfg.manager_low_coverage_pct / 100,
target_seconds: expected_seconds * cfg.manager_target_coverage_pct / 100,
overload_seconds: expected_seconds * cfg.manager_overload_coverage_pct / 100,
work_secs: expected_seconds * 2,
calendar_secs: expected_seconds * 2 + 3600,
work_first: Some(work_start.with_timezone(&Utc)),
work_last: Some(work_end.with_timezone(&Utc)),
is_today: false,
late_start: work_start + TimeDelta::minutes(cfg.manager_late_start_grace_minutes),
early_finish: work_end - TimeDelta::minutes(cfg.manager_early_finish_grace_minutes),
samples_count: 20,
active_samples: 18,
sessions_count: 1,
};
let status = build_employee_operations_status(&cfg, input);
assert_eq!(status["load_status"], "overloaded");
assert_eq!(status["discipline_status"], "off_hours");
assert_eq!(status["data_confidence"], "high");
assert_eq!(status["guardrail"], "evidence_backed_manual_review_only");
}
#[test]
fn employee_operations_low_confidence_blocks_personnel_conclusion() {
let cfg = test_config();
let report_date = NaiveDate::from_ymd_opt(2026, 5, 14).unwrap();
let (work_start, work_end, expected_seconds) = workday_bounds(&cfg, report_date);
let input = EmployeeOperationsInput {
expected_seconds,
low_seconds: expected_seconds * cfg.manager_low_coverage_pct / 100,
target_seconds: expected_seconds * cfg.manager_target_coverage_pct / 100,
overload_seconds: expected_seconds * cfg.manager_overload_coverage_pct / 100,
work_secs: 0,
calendar_secs: 0,
work_first: None,
work_last: None,
is_today: false,
late_start: work_start + TimeDelta::minutes(cfg.manager_late_start_grace_minutes),
early_finish: work_end - TimeDelta::minutes(cfg.manager_early_finish_grace_minutes),
samples_count: 0,
active_samples: 0,
sessions_count: 0,
};
let status = build_employee_operations_status(&cfg, input);
assert_eq!(status["load_status"], "no_data");
assert_eq!(status["idle_status"], "unknown");
assert_eq!(status["data_confidence"], "low");
assert_eq!(status["guardrail"], "low_confidence_not_for_discipline");
assert!(
status["recommended_action"]
.as_str()
.unwrap()
.contains("не использовать как дисциплинарный")
);
}
#[test]
fn operations_summary_counts_attention_groups() {
let rows = vec![
json!({"load_status":"overloaded","idle_status":"no_significant_idle","discipline_status":"ok","data_confidence":"high"}),
json!({"load_status":"underloaded","idle_status":"idle_detected","discipline_status":"late_start","data_confidence":"medium"}),
json!({"load_status":"no_data","idle_status":"unknown","discipline_status":"ok","data_confidence":"low"}),
];
let summary = summarize_operations(&rows);
assert_eq!(summary["status"], "ATTENTION");
assert_eq!(summary["load"]["overloaded_users"], 1);
assert_eq!(summary["load"]["underloaded_users"], 1);
assert_eq!(summary["idle"]["idle_users"], 1);
assert_eq!(summary["discipline"]["review_users"], 1);
assert_eq!(summary["confidence"]["low_users"], 1);
assert_eq!(summary["action_required_users"], 3);
}
#[test]
fn stale_legacy_rdp_sources_are_covered_by_fresh_rust_sources() {
assert!(legacy_rdp_covered_by_rust_sources(
@@ -4386,17 +3888,14 @@ mod tests {
#[test]
fn interpretation_policy_accepts_fraction_thresholds() {
let policy: InterpretationPolicy = serde_json::from_value(json!({
"overload_threshold": 1.15,
"overload_threshold": 0.92,
"underload_threshold": 0.45,
"drop_threshold_pct": 20,
"night_work_after": "20:00",
"weekend_work": true
}))
.unwrap();
assert_eq!(
overload_threshold_to_pct(policy.overload_threshold.unwrap()).round(),
115.0
);
assert_eq!(threshold_to_pct(policy.overload_threshold.unwrap()), 92.0);
assert_eq!(threshold_to_pct(policy.underload_threshold.unwrap()), 45.0);
assert_eq!(threshold_to_pct(policy.drop_threshold_pct.unwrap()), 20.0);
assert_eq!(
+1 -11
View File
@@ -275,20 +275,10 @@ impl AwClient {
) -> Result<()> {
for chunk in events.chunks(chunk_size.max(1)) {
let path = format!("/api/0/buckets/{bucket_id}/events");
self.request_status(Method::POST, &path, Some(json!(chunk)))?;
self.request_json(Method::POST, &path, Some(json!(chunk)), false)?;
}
Ok(())
}
fn request_status(&self, method: Method, path: &str, payload: Option<Value>) -> Result<()> {
let response = self.send_retry(method, path, payload)?;
let status = response.status();
if status.is_success() {
Ok(())
} else {
Err(anyhow!("ActivityWatch {path} returned HTTP {status}"))
}
}
}
fn log(message: &str) {
+16 -6
View File
@@ -38,9 +38,15 @@ ansible-playbook -i inventory.ini deploy_aw_server.yml
Рекомендуемый способ не хранить пароли в репозитории — перед запуском экспортировать их в переменные окружения:
- Linux `aw_server` (SSH пароль root): `AW_SSH_PASSWORD`
- Linux `proxmox` (SSH пароль): `AW_PROXMOX_SSH_PASSWORD`
- Sudo для Linux, если отличается от SSH: `AW_SUDO_PASSWORD` или
`AW_PROXMOX_SUDO_PASSWORD`
- Windows `aw_windows` (WinRM пароль): `AW_WINRM_PASSWORD`
В `group_vars/aw_server.yml` и `group_vars/windows.yml` они читаются через `lookup('env', ...)`.
В `group_vars/aw_server.yml`, `group_vars/proxmox.yml` и
`group_vars/aw_windows.yml` они читаются через `lookup('env', ...)`.
В `ansible/inventory.ini` пароли хранить нельзя: там остаются только host,
user, port и connection-параметры.
## Полный установочный playbook (всё за один запуск)
@@ -125,8 +131,8 @@ Playbook:
- после deploy принудительно запускает `ActivityWatch Recovery` и managed `ActivityWatch Launch *` задачи;
- включает (`Enable-ScheduledTask`) `ActivityWatch Recovery` и managed `ActivityWatch Launch *` задачи перед запуском (иначе WebUI может показывать `Active time: 0s`);
- оставляет `ActivityWatch Recovery` включённым даже при активном `AWatchRusCollectorGuard`: guard является основным контроллером, recovery остаётся fallback/bootstrap path;
- выполняет API smoke-check bucket `aw-watcher-afk_<COMPUTERNAME>` и ожидает свежие события;
- выполняет API smoke-check bucket `aw-watcher-window_<COMPUTERNAME>` и ожидает свежие события (по умолчанию включено);
- выполняет API smoke-check bucket `aw-watcher-afk_<aw_windows_logical_host_id>` и ожидает свежие события;
- выполняет API smoke-check bucket `aw-watcher-window_<aw_windows_logical_host_id>` и ожидает свежие события (по умолчанию включено);
- запускает `validate-deployment.ps1`;
- забирает JSON-отчёт в локальную директорию (`/tmp/aw-rus-validation-<USER>` по умолчанию).
- настраивает scheduled task `ActivityWatch Hayabusa Upload` с периодом и lookback по vars.
@@ -146,8 +152,10 @@ Playbook:
- `aw_windows_legacy_install_root` / `aw_windows_legacy_state_root` — старые production paths, откуда выполняется перенос;
- `aw_windows_migration_report_remote_path` — JSON-отчёт о миграции на Windows-хосте;
- `aw_windows_package_version`, `aw_windows_package_url`, `aw_windows_package_zip_path` — версия и источник Windows-пакета ActivityWatch;
- `aw_windows_api_smoke_check_bucket: ""` — автоматически использовать `aw-watcher-afk_<COMPUTERNAME>`;
- `aw_windows_api_smoke_check_window_enabled: true` — включить дополнительный smoke-check `aw-watcher-window_<COMPUTERNAME>`;
- `aw_windows_domain: ""` — Windows account domain/local logon prefix. Если пусто или `HOST-EXAMPLE`, playbook берёт текущий `$env:COMPUTERNAME` с Windows-хоста через WinRM;
- `aw_windows_logical_host_id: ""` — stable ActivityWatch id для bucket-ов/дашбордов; если пусто, fallback к `COMPUTERNAME`;
- `aw_windows_api_smoke_check_bucket: ""` — автоматически использовать `aw-watcher-afk_<aw_windows_logical_host_id>`;
- `aw_windows_api_smoke_check_window_enabled: true` — включить дополнительный smoke-check `aw-watcher-window_<aw_windows_logical_host_id>` с fallback к физическому `COMPUTERNAME`;
- `aw_windows_api_smoke_check_window_bucket: ""` — переопределить bucket для window smoke-check;
- `aw_windows_api_smoke_check_min_events: 1` — минимум событий, ожидаемых в smoke-check;
- `aw_windows_fail_on_validation_error: true` — завершать playbook ошибкой, если `validate-deployment.ps1` возвращает `overallOk=false`;
@@ -157,7 +165,7 @@ Playbook:
- `aw_windows_hayabusa_auto_upload_hours_back: 6` — lookback для каждого запуска;
- `aw_windows_hayabusa_auto_upload_mode: "incident"` — mode для server-side processing;
- `aw_windows_hayabusa_auto_upload_task_name: "ActivityWatch Hayabusa Upload"` — имя scheduled task.
- `aw_windows_hayabusa_auto_upload_run_as_user: "Администратор"` — production principal для scheduled task на RDP-хосте. На `SHARKON2025` запуск `powershell.exe` из `SYSTEM` возвращал `0xC0000142`, поэтому авто-upload должен идти как interactive/highest task от локального администратора.
- `aw_windows_hayabusa_auto_upload_run_as_user: "Администратор"` — production principal для scheduled task на RDP-хосте. На текущем DetMir RDP-контуре запуск `powershell.exe` из `SYSTEM` возвращал `0xC0000142`, поэтому auto-upload должен идти как interactive/highest task от локального администратора.
## Server-side Hayabusa auto-case и Telegram alerting
@@ -290,3 +298,5 @@ bash scripts/prod_rollout.sh
```
Скрипт попросит `AW_SSH_PASSWORD` и `AW_WINRM_PASSWORD` интерактивно (ввод скрыт) и сложит логи в `.rollout-logs/`.
Для Proxmox можно дополнительно экспортировать `AW_PROXMOX_SSH_PASSWORD`, если
он отличается от `AW_SSH_PASSWORD`.
+42 -5
View File
@@ -47,6 +47,7 @@ aw_worktime_host: "{{ aw_monitored_windows_hostname }}"
aw_rus_health_worktime_api_base: "http://127.0.0.1:5610"
aw_rus_health_state_dir: "{{ aw_server_data_dir }}/health"
aw_rus_health_validation_dir: "{{ aw_rus_health_state_dir }}/windows-validation"
aw_rus_health_rdp_tcp_required: true
aw_browser_smoke_enabled: true
aw_browser_smoke_engine: "chromium-cli"
aw_legacy_db_merge_enabled: false
@@ -58,6 +59,15 @@ aw_hayabusa_telegram_enabled: true
aw_hayabusa_telegram_min_severity: "high"
aw_hayabusa_telegram_bot_token: ""
aw_hayabusa_telegram_chat_ids: ""
aw_security_finding_inbox_enabled: false
aw_security_finding_inbox_required: false
aw_security_finding_inbox_bin: "/usr/local/bin/security-finding-inbox"
aw_security_finding_inbox_min_severity: "medium"
aw_security_finding_executor_work_dir: "{{ aw_server_data_dir }}/security-finding-executor"
aw_security_finding_executor_lock: "/var/lock/aw-security-finding-executor.lock"
aw_containment_engine_bin: "/usr/local/bin/containment-engine"
aw_containment_management_allowlist: ""
aw_containment_blocked_remote_addresses: ""
aw_repo_root: "{{ playbook_dir | dirname }}"
@@ -85,17 +95,44 @@ aw_server_always_active_pattern: "aw-watcher-window"
aw_server_landingpage: "/#/activity/HOST-EXAMPLE/view/"
aw_health_strict_fileops: 0
aw_dlp_policy_engine_enabled: true
aw_dlp_profile: "core_only"
detmir_portal_dlp_profile: "core_only"
detmir_portal_dlp_module_enabled_override: false
aw_dlp_enabled: false
aw_dlp_disabled_reason: "operator_disabled_to_reduce_proxmox_influx_grafana_clickhouse_load"
aw_dlp_disabled_since: ""
aw_dlp_light_collector_enabled: false
aw_dlp_light_guard_enabled: true
aw_dlp_light_guard_load_ratio: "1.50"
aw_dlp_light_guard_mem_available_pct_min: "15"
aw_dlp_light_guard_iowait_pct_max: "20"
aw_dlp_light_guard_strikes_required: 3
aw_dlp_light_guard_state_dir: "{{ aw_server_data_dir }}/health"
aw_dlp_aggregator_bucket_prefixes: "aw-file-operations_,aw-dlp-incidents_"
aw_dlp_aggregator_limit: 500
aw_dlp_aggregator_lookback_hours: 2
aw_dlp_aggregator_overlap_seconds: 60
aw_dlp_aggregator_timeout_seconds: 8
aw_dlp_aggregator_on_calendar: "*:3/15:10"
aw_dlp_aggregator_cpu_quota: "10%"
aw_dlp_aggregator_memory_max: "256M"
aw_containment_enabled: false
aw_containment_mode: "shadow"
aw_containment_policy_path: "/etc/activitywatch/containment-policy.json"
aw_containment_default_ttl_minutes: 60
aw_containment_require_admin_channel_check: true
aw_containment_allow_auto_for_servers: false
aw_dlp_policy_engine_enabled: false
aw_dlp_policy_engine_bind_host: "0.0.0.0"
aw_dlp_policy_engine_port: 5601
aw_dlp_policy_engine_db_path: "{{ aw_server_data_dir }}/dlp-policy-engine.sqlite"
aw_dlp_content_analysis_enabled: true
aw_dlp_integrations_enabled: true
aw_dlp_case_management_enabled: true
aw_dlp_content_analysis_enabled: false
aw_dlp_integrations_enabled: false
aw_dlp_case_management_enabled: false
aw_dlp_case_bind_host: "0.0.0.0"
aw_dlp_case_port: 5602
aw_dlp_case_db_path: "/opt/activitywatch/dlp-case-management/cases.db"
aw_dlp_compliance_enabled: true
aw_dlp_compliance_enabled: false
aw_dlp_compliance_report_dir: "/opt/activitywatch/dlp-compliance/reports"
aw_dlp_compliance_template_path: "/opt/activitywatch/dlp-compliance/templates/152-fz-report.html"
aw_server_post_deploy_health_check_enabled: true
+12
View File
@@ -0,0 +1,12 @@
# Secret handling:
# - put the real Proxmox SSH password into env var before running Ansible:
# export AW_PROXMOX_SSH_PASSWORD='...'
# - if Proxmox and AW server share the same SSH credential, AW_SSH_PASSWORD is
# accepted as a fallback.
ansible_password: "{{ lookup('env', 'AW_PROXMOX_SSH_PASSWORD') | default(lookup('env', 'AW_SSH_PASSWORD'), true) }}"
ansible_become: true
ansible_become_method: sudo
# If sudo password differs, set AW_PROXMOX_SUDO_PASSWORD. Otherwise it reuses
# AW_PROXMOX_SSH_PASSWORD and then AW_SSH_PASSWORD.
ansible_become_password: "{{ lookup('env', 'AW_PROXMOX_SUDO_PASSWORD') | default(lookup('env', 'AW_PROXMOX_SSH_PASSWORD'), true) | default(lookup('env', 'AW_SUDO_PASSWORD'), true) | default(lookup('env', 'AW_SSH_PASSWORD'), true) }}"
+1 -1
View File
@@ -6,7 +6,7 @@ aw-ct ansible_host=10.20.30.13 ansible_user=root ansible_port=22
[aw_windows]
# Примечание: в русифицированных Windows часто нужен "Администратор", а не "Administrator".
win-node1 ansible_host=<WINDOWS_HOST> ansible_user=Администратор ansible_password=CHANGE_ME ansible_connection=winrm ansible_winrm_transport=ntlm ansible_port=5985 ansible_winrm_server_cert_validation=ignore
win-node1 ansible_host=<WINDOWS_HOST> ansible_user=Администратор ansible_connection=winrm ansible_winrm_transport=ntlm ansible_port=5985 ansible_winrm_server_cert_validation=ignore
[aw_pfsense_pollers]
# pfsense-poller1 ansible_host=198.51.100.30 ansible_user=root ansible_port=22
@@ -274,7 +274,7 @@ server {
location ^~ /portal/api/readiness {
proxy_set_header Authorization "";
proxy_set_header X-Remote-User $remote_user;
proxy_pass http://192.0.2.13:8721/api/readiness;
proxy_pass http://127.0.0.1:8720/api/readiness;
proxy_redirect off;
}
+1 -1
View File
@@ -2,7 +2,7 @@
"use strict";
var BAD_HOST = ["HOST", "EXAMPLE"].join("-");
var DEFAULT_HOST = "SHARKON2025";
var DEFAULT_HOST = "HOST-EXAMPLE";
function decode(value) {
try {
@@ -1,5 +1,5 @@
{
"overload_threshold": 1.15,
"overload_threshold": 0.92,
"underload_threshold": 0.45,
"drop_threshold_pct": 20,
"night_work_after": "20:00",
+31 -2
View File
@@ -19,10 +19,16 @@ Forensics усиливают продукт, но не должны перетя
- как меняется загрузка сотрудников и подразделений;
- где тормозят бизнес-процессы.
Канонический контракт по загрузке, простоям, перегрузу, дисциплине процесса и
достоверности данных: [WORKFORCE_OPERATIONS_MODEL_RU.md](WORKFORCE_OPERATIONS_MODEL_RU.md).
Основные KPI:
- UEBA риск: read-only `risk_score/risk_level/reasons` для приоритизации
проверки;
- операционная загрузка: rule-based статусы `load_status`, `idle_status`,
`discipline_status`, `data_confidence` для разбора загрузки, простоев,
перегруза и дисциплины процесса;
- индекс активности: proxy `активное время / плановое рабочее время`;
- взвешенная активность: только при настроенной role/application policy;
- сравнение подразделений за текущий день;
@@ -195,7 +201,7 @@ Worktime API сохраняет daily history как агрегированны
```json
{
"overload_threshold": 0.92,
"overload_threshold": 1.15,
"underload_threshold": 0.45,
"drop_threshold_pct": 20,
"night_work_after": "20:00",
@@ -204,7 +210,9 @@ Worktime API сохраняет daily history как агрегированны
```
`overload_threshold` и `underload_threshold` можно задавать дробью
`0.92`/`0.45` или процентом `92`/`45`; внутри они нормализуются к процентам.
`1.15`/`0.45` или процентом `115`/`45`; внутри они нормализуются к процентам.
Порог перегруза ниже 100% не применяется как перегруз: это защищает отчет от
ложного статуса "перегружен" при обычной высокой занятости.
Если policy-файл отсутствует или отдельное поле не задано, используются
env/default значения:
@@ -260,6 +268,27 @@ env/default значения:
- подтвержденным инцидентом событие становится после регламентной валидации;
- продукт не заявляется как сертифицированная DLP/SIEM/EDR/XDR/СЗИ.
### Runtime boundary для DLP
Актуальный DetMir baseline перед переработкой зафиксирован в
[DETMIR_CURRENT_STATE_RU.md](DETMIR_CURRENT_STATE_RU.md).
Коммерчески и технически DLP нужно подавать как подключаемый модуль, а не как
обязательную часть первого экрана Workforce:
- Workforce должен открываться и строить управленческие показатели без ожидания
DLP evidence, screenshots и heavy correlation;
- DLP endpoint signals, clipboard/USB/print/web/file incidents, evidence review
и screenshots остаются ценным модулем Security/Forensics;
- отключенная или не настроенная DLP должна давать честный disabled-state, а не
ошибку портала;
- тяжелый DLP слой не должен блокировать `/api/operator`, prewarm и первичную
загрузку руководительского/операционного экрана.
Это не означает отказ от DLP-сигналов. Это означает правильную модульность:
ежедневная бизнес-ценность Workforce должна быть доступна быстро, а Security и
Forensics подключаются как углубляющие слои.
## AWatch-rus Forensics
Для разбора сложных событий и пост-инцидентной аналитики.
+7 -46
View File
@@ -55,21 +55,6 @@ Live endpoints, hostnames, tokens and passwords must be supplied through
`/etc/detmir/detmir-check.env` (systemd units) or another private environment
file outside the public repository.
Production note checked on 2026-06-24:
- `DETMIR_PORTAL_URL` must point to the local DetMir portal listener,
currently `http://127.0.0.1:8720`, for server-side health checks. If it is
omitted, `detmir-check` falls back to the public HTTPS gateway and protected
`/readyz`, `/version` and `/metrics` can correctly return `401`, producing a
false operational failure.
- `DETMIR_GATEWAY_HOST=127.0.0.1` is used with the local listener so the Host
header does not accidentally select the public protected gateway path.
- Cold `/api/reports` builds can take more than 60 seconds on the live contour
when cache is empty or concurrent checks are active. The production
`detmir-portal-prewarm.service` therefore uses `curl --max-time 180` and
`TimeoutStartSec=210`. Shorter 45-60 second limits caused false failed
systemd states while the portal eventually returned HTTP 200.
## Текущий планировщик Proxmox, проверено 2026-06-21
На Proxmox уже присутствуют следующие регулярные проверки:
@@ -90,10 +75,10 @@ Production note checked on 2026-06-24:
Наблюдение: отдельный ежедневный полный gate по всей матрице AWatch-rus
отсутствует. Его роль должен закрыть `awatch-contour-daily-check.timer`.
Историческое наблюдение 2026-06-24: `detmir-portal-prewarm.service` был найден
в failed state из-за устаревшего `curl --max-time 60` для холодной сборки
`/api/reports`. В текущей ветке это оформлено как отдельный prewarm/resilience
пакет, а не как обязательная часть DLP production hot path.
Наблюдение: последняя проверка `detmir-portal-prewarm.service` на момент осмотра
имела `Result=exit-code` и `ExecMainStatus=28`. Это не надо маскировать:
канонический check должен показывать такой сбой как fail/warn в зависимости от
политики эксплуатации.
## Матрица требований и проверок
@@ -106,36 +91,14 @@ Production note checked on 2026-06-24:
| Portal hardening | `/healthz`, `/readyz`, `/version`, `/metrics` | `detmir-check` | да | да |
| Windows/RDP | TCP 5985 и 22 | `detmir-check` | да | да |
| ActivityWatch buckets | AFK/window/worktime/session events | `detmir-check` | да | да |
| AWatch DLP buckets | endpoint signals/incidents/review/rules, только если DLP включен | `detmir-check` | условно | условно |
| AWatch DLP health | disabled/core_only должен быть SKIPPED/WARN, `light/full` проверяются через `detmir-dlp` | `detmir-check` | да | да |
| AWatch DLP buckets | endpoint signals/incidents/review/rules | `detmir-check` | да | да |
| AWatch DLP health | remote `dlp-health-check --json` через `detmir-dlp` | `detmir-check` | да | да |
| Grafana evidence | свежий JSON артефакт Grafana check | `detmir-check` | да | да |
| Security events backend | ClickHouse events, если включено | `detmir-check` | да | да |
| Portal contract | role/API smoke | `scripts/awatch-production-hardening-smoke.mjs` | нет | да |
| Pilot contract | demo/API smoke | `scripts/detmir-pilot-demo-smoke.mjs` | нет | да |
| Registry/readiness docs | registry readiness check | `scripts/registry_readiness_check.sh` | опционально | да |
## Timeout/fail-closed параметры live contour
После ручного live-прогона 2026-06-24 production
`/etc/detmir/detmir-check.env` должен содержать bounded timeouts, соответствующие
фактической latency AW datastore:
```env
DETMIR_SERVICE_TIMEOUT_SECONDS=35
DETMIR_BUCKET_TIMEOUT_SECONDS=35
DETMIR_DLP_TIMEOUT_SECONDS=120
DETMIR_CHECK_OVERALL_TIMEOUT_SECONDS=300
```
Назначение:
- не считать bucket `DEAD` только из-за штатной 15-30 секундной latency
большого SQLite datastore;
- не оставлять `detmir-check`, `detmir-dlp`, `ssh` и remote
`dlp-health-check` хвосты при timeout;
- сохранять красный non-zero результат при реальной недоступности, но
завершать проверку bounded.
## Fail-closed политика
Ежедневный check должен завершаться non-zero, если падает обязательная область:
@@ -145,9 +108,7 @@ DETMIR_CHECK_OVERALL_TIMEOUT_SECONDS=300
- Gateway/Portal health;
- RDP/Windows reachability;
- свежесть обязательных bucket streams;
- AWatch DLP health только если DLP runtime включен; при штатном
`AW_DLP_ENABLED=false`/`core_only` disabled-state не является отказом
Workforce/Worktime core;
- AWatch DLP health;
- Grafana evidence freshness.
Event-driven buckets не должны считаться stale только из-за отсутствия новых
+695
View File
@@ -0,0 +1,695 @@
# DetMir/AWatch-rus: hardline resilience hardening
Документ фиксирует реализованные и проверенные шаги по доведению живого
контура до fail-closed уровня. Он не заменяет production runbook; здесь только
изменения, влияющие на отказоустойчивость.
## 2026-06-30: crash-test readiness gate and healthd route boundary
Статус: implemented in repo, deployed on live AW server, verified by manual
crash test.
Что прогонялось:
- baseline `check-aw-full.sh`, SQLite hot-path plan, disk/headroom, RDP guard;
- bounded parallel load на `/api/0/info`,
`/aw-worktime-sessions_SHARKON2025/events?limit=100`,
`aw-detmir-web-category_SHARKON2025` и Worktime API;
- controlled restart: `aw-worktime-api`, `activitywatch-server`,
`AWatchRusCollectorGuard`;
- gateway/ClickHouse/Grafana reachability checks;
- live `scripts/detmir_resilience_check.sh --live` on AW server.
Что найдено:
- `systemctl is-active activitywatch-server` не равен полной готовности API:
сразу после `systemctl restart activitywatch-server` первый `/api/0/info`
мог уйти в 15 секунд timeout, затем API стабилизировался и hot-path
`/events?limit=100` отвечал быстро;
- `aw-dlp-case-management.service` оставался active при disabled DLP profile;
- `aw-rus-healthd.service` падал из-за TCP timeout с AW server до
`192.168.100.19:5985/3389`, хотя фактическая RDP/WinRM проверка с
admin/VPN side и bucket freshness были зелёными;
- SQLite hot-path index
`events_bucketrow_starttime_desc_index` присутствовал и использовался,
`TEMP B-TREE` для worktime event query не строился.
Что изменено:
- `scripts/detmir_resilience_check.sh --live` теперь использует readiness-loop
для `/api/0/info`, проверяет worktime hot path, Worktime API rows/degraded
state, SQLite hot-path index/plan и disabled-state optional DLP/Loki units;
- `aw-rus-healthd-rust` получил fail-closed параметр
`AW_RUS_HEALTH_RDP_TCP_REQUIRED` / `--rdp-tcp-required`;
- default/example остаётся `true`; в DetMir production выставлено
`false`, потому что server-side TCP до RDP сейчас является route/ACL
boundary, а не authoritative proof of collector health;
- active drift `aw-dlp-case-management.service` остановлен, unit оставлен
disabled для штатного будущего включения DLP contour.
Live verification:
- `check-aw-full.sh`: `FRESH=8`, `STALE=0`, `DEAD=0`;
- targeted load after stabilization:
`info_p2/p4/p8/p12` по `24/24` HTTP 200,
worktime events `40/40` HTTP 200,
web category bucket `40/40` HTTP 200,
Worktime today `30/30` HTTP 200;
- `AWatchRusCollectorGuard` restart: service `Running`, `GUARD_CHILDREN=1`,
collector process layout unchanged;
- `aw-rus-healthd.service`: `status=0/SUCCESS`, `ok=11`, `warn=3`, `fail=0`;
- `scripts/detmir_resilience_check.sh --live` after fixes:
readiness, hot path, Worktime API, SQLite index, optional DLP and Loki checks
pass; Hayabusa quarantine warning remains informational evidence to review.
Safety guardrails:
- no AW bucket schema, API, UI or Workforce business logic changed;
- GitHub/Grafana/ClickHouse are still validation/visibility surfaces, not
Russian registry release evidence;
- DLP remains optional/reconnectable, not removed.
## 2026-06-25: optional DLP runtime off switch and statistics
Статус: implemented in repo, deployed, live disable verified on 2026-06-25.
Проблема:
- DLP runtime может создавать избыточную нагрузку на InfluxDB, Grafana,
ClickHouse и AW server при включенных aggregator/exporter/case/report
pipeline;
- простая остановка DLP units раньше приводила бы к ложным красным health,
readiness и contour checks.
Что добавлено:
- `AW_DLP_ENABLED=false` для AW server runtime;
- `DETMIR_DLP_ENABLED=false` для управляющего DetMir contour check;
- `dlp-health-check` возвращает штатный `dlp:mode=disabled`;
- `detmir-dlp` не выполняет SSH health probe при disabled mode;
- `detmir-check`, `check-aw-full`, `check-aw-data` пропускают DLP buckets при
disabled mode;
- `detmir-readiness` не требует DLP Influx write и DLP systemd units при
disabled mode;
- `scripts/detmir_dlp_runtime_control.sh` собирает JSON-срез DLP units/buckets
и выполняет controlled `disable|enable`.
- live `disable` сохраняет отдельные evidence-снимки `current`,
`pre_disable` и `disabled` в
`/var/lib/activitywatch/health/dlp-runtime-history/`.
Safety guardrails:
- ActivityWatch server, worktime, Hayabusa, 1C/ClickHouse core не отключаются;
- historical DLP buckets/evidence не удаляются;
- disabled-state не заявляет, что DLP проверки выполнены;
- это не claim замены DLP/SIEM/EDR и не удаление DLP-функциональности.
Runbook:
- [DLP_OPTIONAL_RUNTIME_RU.md](DLP_OPTIONAL_RUNTIME_RU.md).
Live verification 2026-06-25:
- before disable, active DLP runtime units were present:
`aw-dlp-influx-exporter.timer`, `activitywatch-dlp-aggregator.timer`,
DLP report/integration timers, policy/case services and
`detmir-portal-evidence.service`;
- after disable, active/enabled DLP units: `0/0`;
- `AW_DLP_ENABLED=false`, `AW_DLP_INFLUX_ENABLED=false`,
`AW_DLP_DISABLED_REASON=operator_disabled_to_reduce_influx_grafana_clickhouse_load`;
- `dlp-health-check` and `detmir-dlp` both returned `dlp:mode=disabled`;
- `check-aw-full` reported DLP buckets as `SKIPPED`;
- ActivityWatch core remained active:
`activitywatch-server`, `aw-worktime-api`.
Residual non-DLP findings from the same check:
- RDP-side collectors require separate recovery: AFK/window/worktime buckets
were stale;
- server-side WinRM reachability to `192.168.100.18:5985` was unavailable;
- `aw-rus-healthd.service` was already failed and is tracked separately from
this DLP runtime disable.
## 2026-06-24: Hayabusa poison-package isolation
Статус: implemented locally, unit-tested, deployed on AW server.
Проблема:
- один битый zip в `/opt/hayabusa/inbox/incoming` мог остановить весь
Hayabusa pipeline;
- `aw-hayabusa-drop.path` после повторных падений мог упереться в systemd
start-limit;
- восстановление требовало ручного переноса bad zip в quarantine.
Что изменено:
- `aw-hayabusa-autoprocess-rust` проверяет drop zip до `accept`;
- corrupt/empty/unsafe zip и битые sidecar-файлы не попадают в рабочий inbox;
- bad drop package переносится в `/opt/hayabusa/quarantine/drop/...` вместе с
`.meta.json`, `.caseid`, optional `.sha256` и `reason.json`;
- `aw-hayabusa process-inbox` больше не abort'ит весь batch из-за одного
incoming package;
- failed incoming package переносится в
`/opt/hayabusa/quarantine/incoming/...` с partial staging payload и
`reason.json`, если пакет остался в incoming;
- пакеты, уже архивированные wrapper'ом как `failed-no-evtx` или
`failed-analysis`, остаются в штатном archive/intake manifest для
расследования.
Safety guardrails:
- quarantine не удаляет evidence;
- replay выполняется только после re-export или явного восстановления пакета;
- один poison archive не должен мешать обработке остальных zip;
- operational failures инфраструктуры (`aw-hayabusa` отсутствует, права,
broken runtime) остаются красными и не маскируются как успешная обработка.
Проверки:
```bash
cd /mnt/usb_hdd2/Projects/ActivityWatch-Russian
bash -n aw-server/hayabusa/aw-hayabusa.sh
cd /mnt/usb_hdd2/Projects/ActivityWatch-Russian/adk-rust
export CARGO_TARGET_DIR=/home/igor/.cache/detmir-adk-rust-target
cargo test -p hayabusa-tools
```
Результат проверки:
- `bash -n aw-server/hayabusa/aw-hayabusa.sh` passed;
- `cargo test -p hayabusa-tools` passed: 5 tests passed.
Production deployment note:
- после сборки и доставки нового `aw-hayabusa-autoprocess-rust` нужно
выполнить live dry-run на empty drop и контролируемый bad-zip test в
непроизводственном каталоге или с временным isolated `--drop-dir`;
- live production queue руками не мутировать без предварительного backup/listing.
Live rollout evidence:
- deployed on AW server on `2026-06-24`;
- previous `/usr/local/bin/aw-hayabusa` and
`/usr/local/bin/aw-hayabusa-autoprocess-rust` were backed up with timestamp
suffix;
- `/usr/local/bin/aw-hayabusa doctor` returned OK;
- isolated empty-drop dry-run returned `no zip packages in drop dir`;
- production queue after rollout: `incoming_zip=0`, `DROP_COUNT=0`,
`aw-hayabusa-drop.path=active`, `aw-hayabusa-drop.service=inactive`;
- stale staging residue from `2026-06-20` was moved, not deleted, to
`/opt/hayabusa/quarantine/staging-stale-20260624T190739Z/` with
`reason.json`;
- after cleanup: `staged_dirs=0`, `archived_packages=74`,
`archived_payloads=74`.
## 2026-06-24: Windows collector guard service child watchdog
Статус: implemented locally, static checks passed, deployed on RDP host.
Проблема:
- Windows service `AWatchRusCollectorGuard` мог оставаться в состоянии
`running`, когда дочерний `aw-windows-telemetry.exe collector-guard` уже
отсутствовал;
- SCM recovery не срабатывал, потому что сам service wrapper не падал.
Что изменено:
- `AWatchRusCollectorGuardService.cs` теперь подписывается на `Process.Exited`;
- при неожиданном выходе child-процесса wrapper делает bounded restart;
- restart budget: 5 child restarts за 600 секунд, задержка 5 секунд;
- при исчерпании бюджета wrapper завершает service с ошибкой, чтобы Windows
Service Control Manager применил recovery actions;
- installer включает `sc.exe failureflag <service> 1`, чтобы recovery actions
применялись к service failures, а не только к crash-путям;
- `ActivityWatch Recovery` остаётся fallback/bootstrap задачей и не отключается.
Safety guardrails:
- штатный `Stop-Service`/shutdown выставляет `stopping=true`, поэтому child exit
во время остановки не считается аварией;
- wrapper не меняет collector mode, bucket names, event schema и AW API;
- service-level recovery ограничен существующим `sc.exe failure` budget.
Проверки:
```powershell
pwsh -NoProfile -Command '<compile AWatchRusCollectorGuardService.cs through Add-Type>'
pwsh -NoProfile -Command '<parse install-collector-guard-service.ps1>'
powershell -NoProfile -ExecutionPolicy Bypass -File windows/install-collector-guard-service.ps1
Get-Service AWatchRusCollectorGuard
Get-Content C:\ProgramData\AWatch-rus\logs\collector-guard-service.log -Tail 20
```
Результат локальной проверки:
- `AWatchRusCollectorGuardService.cs` compiled through PowerShell `Add-Type`;
- `windows/install-collector-guard-service.ps1` parsed with PowerShell parser;
- live install/restart validation still requires Windows/RDP deployment window.
Live rollout evidence:
- deployed on RDP host on `2026-06-24`;
- previous service source, installer and exe were backed up under
`C:\ProgramData\AWatch-rus\backup\collector-guard-service-20260624T190827Z`;
- installer completed with `Runtime: rust`, `Mode: enforce`;
- SCM `failureflag` is enabled:
`FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE`;
- controlled fault-injection killed only the child
`aw-windows-telemetry.exe collector-guard`;
- wrapper observed child exit, attempted bounded restarts, SCM recovery restarted
wrapper after budget exhaustion, and child stabilized;
- final validation after one guard loop: service `Running`, `CHILD_COUNT=1`,
child `aw-windows-telemetry.exe`, latest rust guard cycle `status=ok`.
Live validation после деплоя:
- убить только child `aw-windows-telemetry.exe collector-guard`;
- убедиться, что service остаётся running и child перезапущен;
- повторить child crash больше 5 раз за 600 секунд в тестовом окне;
- убедиться, что service перешёл через SCM recovery, а не остался
`running/no child`.
## 2026-06-24: contour resilience check
Статус: implemented locally, shell syntax/repo-mode passed, live AW server
check passed.
Проблема:
- отдельные исправления легко потерять при deploy/drift;
- CI не проверял, что poison-package isolation и child watchdog реально
присутствуют в коде и документации;
- live-проверки должны оставаться read-only и не маскировать production сбой.
Что добавлено:
- `scripts/detmir_resilience_check.sh`;
- `--repo` режим для CI-safe проверки hardening-файлов, паттернов и docs;
- `--live` режим для read-only проверки локального AW/Hayabusa host:
`activitywatch-server`, `aw-worktime-api`, AW `/api/0/info`, failed systemd
units, Hayabusa `incoming/drop/quarantine`, SQLite DB/WAL size;
- `RUN_RESILIENCE_CHECK=1` hook в `scripts/run_awatch_contour_check.sh`;
- `DETMIR_RESILIENCE_STRICT_SECRETS=1` режим, который fail'ит literal
`ansible_password`/`ansible_become_password` в private inventory без вывода
значений.
Safety guardrails:
- check не рестартует сервисы, не двигает очереди, не пишет в production dirs;
- secret check печатает только факт наличия literal assignments, не значения;
- live mode запускается явно через `--live` или `--all`;
- GitHub/public CI может использовать только `--repo`.
Secret handling update 2026-06-30:
- literal `ansible_password` и `ansible_become_password` удалены из локального
`ansible/inventory.ini`;
- `aw_server` читает SSH/sudo secrets из `AW_SSH_PASSWORD` и
`AW_SUDO_PASSWORD`;
- `proxmox` читает SSH/sudo secrets из `AW_PROXMOX_SSH_PASSWORD` и
`AW_PROXMOX_SUDO_PASSWORD`, с fallback на `AW_SSH_PASSWORD` /
`AW_SUDO_PASSWORD`;
- `aw_windows` читает WinRM secret из `AW_WINRM_PASSWORD`;
- `inventory.example.ini` больше не содержит placeholder password-поля.
Проверки:
```bash
bash -n scripts/detmir_resilience_check.sh
bash scripts/detmir_resilience_check.sh --repo
```
Результат локальной проверки:
- shell syntax passed for `scripts/detmir_resilience_check.sh`;
- shell syntax passed for `scripts/run_awatch_contour_check.sh`;
- repo-mode passed with `ok=15`, `fail=0`;
- repo-mode reported one WARN: literal Ansible password assignments appear to
exist in `ansible/inventory.ini`; values are not printed, and
`DETMIR_RESILIENCE_STRICT_SECRETS=1` converts this to fail for private
contour gates.
Live AW server result:
- `bash /tmp/detmir_resilience_check.sh --live` passed on AW server;
- result: `ok=9`, `warn=1`, `fail=0`;
- WARN is expected after this rollout: one quarantine `reason.json` exists for
the moved stale Hayabusa staging residue.
## 2026-06-24: live drift fixes after full contour re-check
Статус: deployed and verified live.
Что было найдено:
- ClickHouse container was healthy by direct SQL checks, but Docker Compose did
not define a container `HEALTHCHECK`; because of that
`aw-1c-clickhouse-health.service` failed with
`docker healthcheck is not configured`.
- `detmir-auto` used the default public gateway URL for portal checks when
`/etc/detmir/detmir-check.env` did not set `DETMIR_PORTAL_URL`; protected
public `/readyz`, `/version` and `/metrics` returned legitimate `401`.
- `detmir-portal-prewarm.service` had `curl --max-time 60`, but a cold
`/api/reports` build on the live contour can take more than 60 seconds.
- `aw-rus-healthd-rust` used the default 20 second wrapper timeout; under
concurrent checks `dlp-health-check --json` could be killed mid-output and be
reported as `invalid JSON output`.
Что изменено:
- `clickhouse-1c/docker-compose.yml` now defines a ClickHouse client
`HEALTHCHECK`, deployed to `/opt/activitywatch/clickhouse-1c/docker-compose.yml`;
- production `/etc/detmir/detmir-check.env` now contains
`DETMIR_PORTAL_URL=http://127.0.0.1:8720` and
`DETMIR_GATEWAY_HOST=127.0.0.1`;
- `ops/systemd/detmir-portal-prewarm.service` is now tracked in the repo and
deployed with `curl --max-time 180` and `TimeoutStartSec=210`;
- production `/etc/activitywatch/aw-server.env` and
`aw-server/aw-server.env.example` now set
`AW_RUS_HEALTH_WRAPPER_TIMEOUT_SECONDS=90`.
Verification:
- `check-aw-full.sh`: `FRESH=8`, `STALE=0`, `DEAD=0`;
- `detmir-check` through the production env file: `ok=true`,
`service_failures=0`;
- `detmir-auto.service`, `awatch-contour-daily-check.service`,
`awatch-contour-weekly-check.service`: `status=0/SUCCESS`;
- `detmir-portal-prewarm.service`: `status=0/SUCCESS`;
- `aw-1c-clickhouse-health.service`: `status=0/SUCCESS`, Docker state
`(healthy)`;
- `aw-rus-healthd.service`: `status=0/SUCCESS`, failed systemd units on
AW server and Proxmox are zero.
## 2026-06-25: portal cold-start prewarm after service restart
Статус: deployed and verified live at the time, then superseded by the
fail-soft hot-path boundary below.
Что было найдено:
- после ручного `systemctl restart detmir-portal` первый
`/api/reports?role=manager` может выполнять холодный расчет дольше 120 секунд;
- `detmir-portal-prewarm.timer` держит cache теплым каждые 30 минут, но не
запускается немедленно при ручном рестарте портала.
- одного prewarm недостаточно, если каждый пользовательский `/api/reports`
заново запускает тяжелую генерацию отчета.
Что изменено:
- добавлен tracked drop-in
`ops/systemd/detmir-portal.service.d/30-prewarm-after-start.conf`;
- production drop-in `/etc/systemd/system/detmir-portal.service.d/30-prewarm-after-start.conf`
запускает `detmir-portal-prewarm.service` через `systemctl --no-block` после
каждого старта портала;
- prewarm остается best-effort: портал стартует независимо, а тяжелый
`/api/reports` прогревается в фоне.
- в `detmir-portal` добавлен short-lived in-process report cache с TTL 120
секунд и защитой от stampede: первый report-запрос строит payload, следующие
report endpoints в окне TTL отдают тот же payload без повторной генерации.
- в `/metrics` добавлены отдельные счетчики report-cache:
`awatch_report_requests_total`, `awatch_report_cache_hits_total`,
`awatch_report_cache_misses_total`. Старый
`awatch_reports_generated_total` остается счетчиком успешно завершенных
тяжелых генераций отчета, а не счетчиком HTTP-запросов.
Verification:
- `/healthz`: `200`;
- `/readyz`: `200`, `status=ready`;
- prewarm after restart: `status=0/SUCCESS`;
- cold `/api/reports?role=manager` after expired cache: `200`, около `63s`;
- warm `/api/reports?role=manager`: `200`, около `0.34..0.35s` в трех
последовательных запросах;
- `awatch_reports_generated_total` не вырос после трех warm report-запросов;
- `awatch_report_requests_total` растет на report endpoints, а
`awatch_report_cache_hits_total` растет на warm cache-запросах;
- во время cold/prewarm сборки `awatch_report_requests_total` показывает
входящие report-запросы до ожидания cache lock, а
`awatch_reports_generated_total` растет только после готового payload;
- `workforce_operations.summary` и `workforce_operations.rows` доступны в JSON;
- browser smoke: блок `Операционная загрузка` отрисован, строки сотрудников
видны, console errors/warnings отсутствуют.
Superseded note:
- restart-triggered external prewarm reduced warm-cache latency, but it also
kept a heavy full-report job coupled to service restart;
- after the DLP/hot-path phase 1 change, the preferred production behavior is
immediate `warming`/`STALE` API response from the portal itself, not a
mandatory heavy `ExecStartPost` prewarm after every restart;
- legacy drop-ins
`/etc/systemd/system/detmir-portal.service.d/20-prod-timeout.conf` and
`/etc/systemd/system/detmir-portal.service.d/30-prewarm-after-start.conf`
are now treated as stale deployment residue and are removed by
`ansible/deploy_detmir_portal.yml`.
## 2026-06-25: current state and first DLP hot-path boundary
Статус: phase 1 implemented, targeted Rust tests passed, deployed once on the
DetMir portal host and API-smoke verified. Follow-up production cleanup of stale
restart-prewarm drop-ins is pending until DetMir VPN handshake is stable again.
Фактический runtime:
- production `detmir-portal` binary:
`653b22b0fbf29a22f7de42ade7b689490b1de16fa07e785e4e0efd3078e7a3bc`;
- deploy command used:
`ansible-playbook -i inventory.ini deploy_detmir_portal.yml --limit proxmox -e detmir_portal_bind_override=0.0.0.0:8720 -e detmir_portal_dlp_module_enabled_override=false`;
- `/healthz`: `status=ok` after deploy;
- `/readyz`: `status=ready` after deploy;
- `/api/reports`: `ok=true`, `cache_status=warming`,
`modules.dlp.enabled=false`, `modules.dlp.hot_path=false`;
- `/api/operator`: `cache_status=warming`, `summary.severity=STALE`,
`modules.dlp.status=disabled`, `incidents=0`;
- server log: `/api/operator` returned `200` with `latency_ms=49`;
- browser smoke after restart: `loadStatus=STALE`, progress `100%`,
`LOADING=false`, `EMPTY=false`, `ERROR=false`.
Что это означает:
- первичное зависание портала устранено на уровне UX/cache/stale fallback;
- `/api/operator` no longer waits for the cold full snapshot and can return a
bounded `warming` payload;
- тяжелая генерация полного отчета/snapshot все еще может быть дорогой во время
cold/prewarm;
- DLP/security enrichment has a first runtime boundary out of the Workforce hot
path: phase 1 used `DETMIR_PORTAL_DLP_MODULE_ENABLED=false`; the current
DetMir production default keeps DLP runtime disabled/`core_only`, while
`light` remains an explicit operator re-enable profile after resource check;
- текущее состояние зафиксировано отдельно:
`docs/DETMIR_CURRENT_STATE_RU.md`.
Архитектурное решение для следующего шага:
- Workforce core должен оставаться быстрым и доступным без DLP;
- DLP evidence, endpoint signals, screenshots, case review, heavy correlation
and forensics enrichment должны стать optional module;
- prewarm не должен обязательно выполнять heavy DLP path;
- Security/Forensics views при отключенной DLP должны показывать disabled-state,
а не ломать portal readiness.
Реализованная первая граница:
- CLI/env flag: `--dlp-module-enabled` /
`DETMIR_PORTAL_DLP_MODULE_ENABLED`;
- DetMir production default после resource hardening: `false` / `core_only`,
чтобы обычный deploy/recovery не возвращал DLP нагрузку на Proxmox, AW,
ClickHouse, InfluxDB и Grafana;
- `light` допускается только как явное operator re-enable действие после
resource check; при `light` основной portal snapshot не читает тяжелые
incident/case/review/audit DLP state, а evidence/case/exporter path остается
выключенным;
- Ansible deploy parameter:
`detmir_portal_dlp_module_enabled_override`.
Проверено локально:
- `cargo test -p detmir-portal --locked`;
- `cargo clippy -p detmir-portal --all-targets --locked -- -D warnings`.
Deployment cleanup status:
- `ansible/deploy_detmir_portal.yml` now removes stale restart-prewarm drop-ins:
`20-prod-timeout.conf` and `30-prewarm-after-start.conf`;
- repeat production deploy of that cleanup is pending because the DetMir
`pfSense-gate-UDP4-1194-vpn_prog10-config` tunnel later failed TLS handshake
to `178.178.98.83:1194`;
- do not claim final production prewarm cleanup until
`systemctl cat detmir-portal` no longer shows `ExecStartPost` prewarm.
Ограничения:
- это не удаление DLP collectors;
- это не claim, что production DLP decoupling уже завершен без live smoke;
- это не registry release evidence;
- GitHub/GitHub Actions не являются primary registry build contour.
## 2026-06-30: DLP disabled/core_only default, load guard and rollback
Статус: implemented in repository defaults/scripts/docs, deployed on live
DetMir contour and verified manually.
Что изменено:
- DetMir production defaults переведены в `AW_DLP_ENABLED=false` и
`AW_DLP_PROFILE=core_only`;
- `aw_dlp_enabled=false`, `aw_dlp_influx_enabled=false`;
- lightweight collector остается подключаемым, но не стартует по умолчанию;
- heavy DLP component flags в production defaults остаются выключены;
- `detmir_portal_dlp_module_enabled_override=false` показывает честный
disabled-state в портале без heavy evidence/case path;
- добавлен `scripts/detmir_dlp_load_guard.sh`;
- `detmir-dlp-load-guard.timer` контролирует load/RAM/iowait и при перегрузе
переводит DLP в `core_only`;
- Ansible DLP tasks больше не имеют DLP-heavy default `true`;
- `scripts/detmir_dlp_runtime_control.sh` получил профили
`core_only`, `light`, `on_demand`, `full`;
- перед каждым `set-profile` сохраняется rollback-снимок systemd
active/enabled состояния DLP units;
- `rollback` восстанавливает предыдущее состояние DLP units без изменения
retention и без запуска Loki CT.
Эксплуатационная позиция:
- Loki CT отключён намеренно для снижения нагрузки на Proxmox VM/LXC;
- Loki не является обязательной зависимостью Workforce/Worktime/AW core;
- DLP не удалён: lightweight-сбор нужен для UEBA, а тяжелый runtime не должен
возвращаться обычным deploy/recovery;
- Hayabusa/Velociraptor findings остаются отдельным optional security layer
через Security Finding Inbox / ClickHouse и не требуют Loki.
Runbook:
- `docs/DLP_RESOURCE_PROFILES_RU.md`;
- `docs/DLP_OPTIONAL_RUNTIME_RU.md`.
## 2026-06-24: fail-closed timeout hardening after manual live run
Статус: implemented locally, targeted Rust tests passed, deployed and verified
live.
Что было найдено ручным прогоном:
- при деградации `activitywatch-server` запросы `/api/0/buckets` и отдельные
bucket event endpoints могли занимать 15-30 секунд;
- `detmir-check` мог зависнуть без общего дедлайна, а штатные
daily/weekly checks оставались в `activating`;
- timeout в `detmir-check` убивал shell, но мог оставить `detmir-dlp`/`ssh`
хвост, который удерживал stdout pipe;
- `detmir-dlp` не имел собственного SSH timeout;
- прямой `dlp-health-check --json` на AW server мог зависать дольше ожиданий;
- `aw-worktime-autoheal-rust` считал ошибкой timeout чтения response body после
POST backfill, хотя ActivityWatch уже мог применить запись.
Что изменено:
- `detmir-check` получил общий watchdog
`DETMIR_CHECK_OVERALL_TIMEOUT_SECONDS` и env-настройки
`DETMIR_SERVICE_TIMEOUT_SECONDS`, `DETMIR_BUCKET_TIMEOUT_SECONDS`,
`DETMIR_DLP_TIMEOUT_SECONDS`;
- production `/etc/detmir/detmir-check.env` настроен на:
`DETMIR_SERVICE_TIMEOUT_SECONDS=35`,
`DETMIR_BUCKET_TIMEOUT_SECONDS=35`,
`DETMIR_DLP_TIMEOUT_SECONDS=120`,
`DETMIR_CHECK_OVERALL_TIMEOUT_SECONDS=300`;
- `detmir-check` и `detmir-auto` убивают timed-out child process group, чтобы
не оставлять shell/SSH/DLP хвосты;
- `detmir-dlp` получил bounded SSH timeout и больше не выносит SSH child в
отдельную process group, чтобы parent timeout мог убить всю ветку;
- `dlp-health-check` получил общий self-timeout
`AW_DLP_HEALTH_OVERALL_TIMEOUT_SECONDS` с default 120 секунд;
- `aw-worktime-autoheal-rust` для POST events проверяет HTTP status и не читает
response body, потому что body не нужен для backfill evidence.
Safety guardrails:
- изменения не меняют AW API schema, bucket names, UI или product workflow;
- timeout failure остается красным, но больше не оставляет активные процессы и
lock poisoning;
- production timeouts расширены только до фактической live latency, общий
deadline остается bounded;
- remote DLP и worktime autoheal не публикуют secrets/PII в logs сверх уже
существующих operational identifiers.
Проверки:
```bash
cargo test --manifest-path adk-rust/Cargo.toml \
-p detmir-check -p detmir-auto -p detmir-dlp -p dlp-health-check \
-p worktime-autoheal
cargo build --manifest-path adk-rust/Cargo.toml --release \
-p detmir-check -p detmir-auto -p detmir-dlp -p dlp-health-check \
-p worktime-autoheal
```
Live verification:
- `dlp-health-check --json`: `ok=22`, `warn=0`, `fail=0`, elapsed 17s;
- `aw-worktime-autoheal.service`: success, posted `afk=28`, `win=28`;
- `aw-rus-healthd.service`: success, `ok=13`, `warn=1`, `fail=0`;
- `detmir-check` through production env: `rc=0`, elapsed 19s;
- `detmir-auto.service`: `rc=0`, elapsed 56s, bucket `dead=0`, `stale=0`,
`ok=8`;
- `awatch-contour-daily-check.service`: `rc=0`, elapsed 14s;
- `awatch-contour-weekly-check.service`: `rc=0`, elapsed 136s;
- `check-aw-full.sh`: `FRESH=8`, `STALE=0`, `DEAD=0`;
- final AW/PVE failed systemd units: `0`;
- final RDP guard: service `Running`, 13 telemetry processes, last guard cycles
`status=ok problems=0`.
## 2026-06-30: manual collection and analysis smoke after DLP light enablement
Статус: partially green, live fixes applied, one external reachability blocker
remains.
Ручной прогон подтвердил:
- `activitywatch-server` and `aw-worktime-api` are active;
- Worktime API `/reports/worktime/today` returns current data for 4 users;
- DetMir portal `/portal` renders through browser and no frontend console error
was observed for the tested portal pages;
- `/api/manager` returns OK after restoring the portal timeout to 25 seconds;
- `/api/operator` returns current collection/DLP/Grafana/1C/worktime blocks;
- Security Finding Inbox is reachable for `security`/`admin` role headers and
returns ClickHouse backend `status=ok`, `open_count=0`;
- DLP light warehouse is present on the portal host and DLP checks show OK in
the operator card;
- Grafana backend is reachable directly at `10.10.10.11:3000/api/health`.
Live fixes applied:
- removed stale systemd drop-in
`/etc/systemd/system/detmir-portal.service.d/10-detmir-check-env.conf`;
- restored `/etc/detmir-portal.env` timeout to
`DETMIR_PORTAL_TIMEOUT_SECONDS=25`;
- updated `/etc/detmir/detmir-check.env` for the current light profile:
`DETMIR_DLP_ENABLED=true` and `DETMIR_DISABLE_DLP_HEALTH_CHECK=true`;
- updated `/var/lib/detmir-ai/latest-run` to the fresh 2026-06-30
`detmir-check` JSON so the portal no longer displays the stale 2026-06-25
collection snapshot;
- updated Ansible to remove the stale portal timeout override during deploy.
Remaining live blocker:
- `detmir-check` still fails by design because RDP host `192.168.100.19`
responds to ICMP, but TCP `22` and `5985` time out from the DetMir contour;
- `awatch-contour-daily-check.service` therefore remains failed with
`service_failures=2`;
- this is not an AW-server, Worktime API, DLP warehouse, ClickHouse, or portal
rendering failure. It is the current RDP control/reachability failure.
Grafana note:
- Browser access to Grafana through the gateway is protected by Basic Auth and
the current certificate chain is not trusted by the Playwright browser when
opened by IP/DNS in this run;
- direct backend health check is green:
`http://10.10.10.11:3000/api/health -> 200`;
- gateway returns `401` without credentials, which is expected for the
protected dashboard entrypoint.
+10
View File
@@ -4,6 +4,12 @@ Explainable Workforce KPI отвечает на вопрос: почему по
активности. Слой предназначен для руководителя, ИБ и администратора, но не
является HR-оценкой сотрудника и не использует ML/LLM.
Смежный, но отдельный contract по операционной загрузке описан в
[WORKFORCE_OPERATIONS_MODEL_RU.md](WORKFORCE_OPERATIONS_MODEL_RU.md). KPI
объясняет процент активности, а Workforce Operations показывает загрузку,
простои, перегруз, дисциплину процесса, достоверность данных и рекомендуемое
ручное действие.
## API
Endpoint:
@@ -102,6 +108,10 @@ Security и Forensics не получают Workforce Dashboard через `/api
Раздел содержит KPI score, confidence, coverage, факторы, warnings и
рекомендации.
Блок `Операционная загрузка` в портале использует другой payload:
`workforce_operations`. Он не заменяет explainable KPI и не должен
интерпретироваться как автоматическая HR-оценка.
## Ограничения Pilot v1
- Это не ML и не LLM.
+5
View File
@@ -83,6 +83,11 @@ gateway:
/d/detmir-rdp-user-activity/detmir3a-rabota-pol-zovatelej-v-rdp?orgId=1&from=now-7d&to=now&timezone=browser&var-host=SHARKON2025&refresh=5m
```
`var-host=SHARKON2025` здесь является stable logical host id, а не требованием
к физическому Windows `COMPUTERNAME`. При переименовании RDP-сервера dashboard
должен продолжать смотреть на тот же logical id до отдельной planned migration.
См. `docs/WINDOWS_LOGICAL_HOST_ID_RU.md`.
В портале он доступен как кнопка `Графики сотрудников`.
Не включайте `[auth.anonymous]` для решения этой задачи: это откроет Grafana на
-161
View File
@@ -7,20 +7,6 @@ ClickHouse не является обязательной зависимость
перезапускайте ClickHouse для восстановления отчетов рабочего времени, если нет
отдельного подтвержденного отказа ClickHouse.
## Stable host id
Worktime reports используют stable logical host id, а не обязательно текущее
Windows `COMPUTERNAME`. Для DetMir production текущий logical id:
```text
SHARKON2025
```
При переименовании RDP-сервера не меняйте `awHostname` автоматически. Сначала
обновите Windows account domain для задач, затем проверьте, что collectors
продолжают писать в bucket-и `*_SHARKON2025`. Подробный порядок:
`docs/WINDOWS_LOGICAL_HOST_ID_RU.md`.
## Симптомы перегруза
- `/portal/api/reports?role=executive` открывается медленно или отвечает
@@ -240,153 +226,6 @@ curl -sS --max-time 8 http://<PORTAL_HOST>/portal/api/health | jq
curl -sS --max-time 12 "http://<PORTAL_HOST>/portal/api/reports?role=executive" | jq '.status'
```
## Production repair: AW SQLite hot path, 2026-06-30
Симптомы:
- `activitywatch-server` отвечает `503` на bucket API;
- журнал содержит `poisoned lock` / `database is locked`;
- `aw-worktime-api` уходит в bounded `DEGRADED`;
- `/buckets/aw-worktime-sessions_<HOST>/events?limit=...` тайм-аутится даже
при малом лимите;
- RDP browser/category collector пишет `bucket create failed` или timeout.
Порядок безопасного восстановления:
1. Остановить RDP guard и процессы `aw-windows-telemetry`, чтобы не продолжать
штурмовать AW API.
2. Остановить `aw-worktime-*` timers/services и другие локальные потребители AW
API.
3. Перезапустить `activitywatch-server` отдельно и проверить `/api/0/info`.
4. Если bucket metadata отвечает, но `/events` медленный, проверить SQLite plan:
```sql
EXPLAIN QUERY PLAN
SELECT id,starttime,endtime,data
FROM events
WHERE bucketrow=(SELECT id FROM buckets WHERE name='aw-worktime-sessions_<HOST>')
ORDER BY starttime DESC
LIMIT 100;
```
Если план строит `TEMP B-TREE FOR ORDER BY`, нужен составной индекс:
```sql
CREATE INDEX IF NOT EXISTS events_bucketrow_starttime_desc_index
ON events(bucketrow, starttime DESC);
ANALYZE;
PRAGMA optimize;
PRAGMA integrity_check;
```
Индекс добавлять только в controlled window:
- остановить `activitywatch-server`;
- сделать rollback backup SQLite DB;
- создать индекс;
- проверить `PRAGMA integrity_check = ok`;
- запустить `activitywatch-server`;
- проверить, что `/events?limit=100` больше не тайм-аутится.
Production DetMir repair 2026-06-30:
- оставлены две свежие ежедневные SQLite VACUUM backup-копии, старые backup-и
ротированы для освобождения места;
- создан rollback backup:
`/var/lib/activitywatch/backups/db/aw-sqlite-before-hotpath-index-20260630T035032Z.db`;
- добавлен индекс `events_bucketrow_starttime_desc_index`;
- `ROCKET_WORKERS=8` добавлен в `/etc/activitywatch/aw-server.env`;
- для `aw-worktime-api.service` добавлен stabilization drop-in:
`AW_WORKTIME_EVENTS_LIMIT=100`,
`AW_WORKTIME_AW_HTTP_TIMEOUT_SECONDS=25`,
`AW_WORKTIME_EVENTS_CACHE_TTL_SECONDS=600`,
`AW_WORKTIME_REPORT_STALE_TTL_SECONDS=7200`.
После ремонта проверить:
```bash
curl -sS --max-time 10 http://127.0.0.1:5600/api/0/info
curl -sS --max-time 15 \
'http://127.0.0.1:5600/api/0/buckets/aw-worktime-sessions_SHARKON2025/events?limit=100'
curl -sS --max-time 15 \
'http://127.0.0.1:5610/reports/worktime/today?format=json' | jq '{rows:(.rows|length),degraded,runtime}'
```
Также проверить отсутствие новых `poisoned lock` после финального старта:
```bash
journalctl -u activitywatch-server --since '<FINAL_START_TIME>' --no-pager |
grep -E 'poisoned lock|Taking datastore lock failed|database is locked'
```
## Crash/readiness test after AW repair
Цель: проверить, что контур выдерживает restart и короткую параллельную
нагрузку, а проверки не путают `systemctl active` с готовым API.
Порядок:
1. Зафиксировать baseline:
```bash
./check-aw-full.sh
```
2. На AW server проверить readiness, hot-path и Worktime API:
```bash
AW_API=http://127.0.0.1:5600 \
AW_WORKTIME_API=http://127.0.0.1:5610 \
AW_LOGICAL_HOST_ID=SHARKON2025 \
scripts/detmir_resilience_check.sh --live
```
Если скрипт запускается с ноутбука, live-mode нужно выполнять на самом
AW-сервере через SSH/Ansible, потому что он проверяет local systemd и SQLite.
3. Controlled restart:
```bash
systemctl restart aw-worktime-api
curl -sS --max-time 12 \
'http://127.0.0.1:5610/reports/worktime/today?format=json&host=SHARKON2025&allow_stale=1' |
jq '{rows:(.rows|length),degraded}'
systemctl restart activitywatch-server
# Не считать "active" готовностью: дождаться HTTP readiness.
timeout 90 bash -c 'until curl -fsS --max-time 8 http://127.0.0.1:5600/api/0/info >/dev/null; do sleep 2; done'
curl -sS --max-time 15 \
'http://127.0.0.1:5600/api/0/buckets/aw-worktime-sessions_SHARKON2025/events?limit=100' >/dev/null
```
4. Проверить, что после рестарта нет новых lock/503:
```bash
journalctl -u activitywatch-server --since '<RESTART_TIME>' --no-pager |
grep -E 'poisoned lock|Taking datastore lock failed|database is locked|503'
```
5. Проверить RDP guard restart отдельно:
```powershell
Restart-Service AWatchRusCollectorGuard -Force
Start-Sleep -Seconds 75
Get-Service AWatchRusCollectorGuard
Get-Process aw-windows-telemetry -ErrorAction SilentlyContinue | Measure-Object
```
Ожидаемый результат для DetMir после ремонта 2026-06-30:
- `check-aw-full.sh`: `FRESH=8`, `STALE=0`, `DEAD=0`;
- `/events?limit=100` отвечает за bounded time и использует
`events_bucketrow_starttime_desc_index`;
- `aw-rus-healthd.service` завершается `status=0/SUCCESS`;
- server-side TCP до RDP может быть `warn`, если
`AW_RUS_HEALTH_RDP_TCP_REQUIRED=false`, но bucket freshness и WinRM/SSH
через admin path должны оставаться зелёными;
- optional DLP/Loki heavy runtime units должны быть inactive в экономном
production profile.
## Rollback
Rollback нужен, если после обновления бинарника или env-настроек:
+2
View File
@@ -19,6 +19,8 @@
## Workforce сценарий
- Отображается индекс активности и объяснение факторов.
- Отображается блок `Операционная загрузка`: загрузка, простой, перегруз,
дисциплина процесса и достоверность данных.
- Доступно сравнение подразделений и ответственных.
- Видны тренды daily, weekly и monthly, если данные есть.
- Отдельно показываются признаки перегрузки и недозагрузки.
+45
View File
@@ -46,6 +46,51 @@ React/Tauri-интерфейса без переписывания backend-ло
- `GET /api/readiness/latest` - готовность системы;
- `GET /api/workforce/policy/explain` - объяснение расчёта показателей.
`GET /api/reports` должен сохранять additive payload `workforce_operations`.
Это основной contract для экрана руководителя по загрузке, простоям, перегрузу,
дисциплине процесса и достоверности данных. Клиент должен читать:
- `workforce_operations.summary`;
- `workforce_operations.rows`;
- `workforce_operations.model`;
- `workforce_operations.rows[].load_status`;
- `workforce_operations.rows[].idle_status`;
- `workforce_operations.rows[].discipline_status`;
- `workforce_operations.rows[].data_confidence`;
- `workforce_operations.rows[].recommended_action`.
Подробная семантика статусов:
[WORKFORCE_OPERATIONS_MODEL_RU.md](WORKFORCE_OPERATIONS_MODEL_RU.md).
`GET /api/reports` также публикует additive payload `modules.dlp`.
Клиент должен трактовать его как runtime capability, а не как claim
сертифицированной DLP:
- `modules.dlp.enabled`;
- `modules.dlp.status`;
- `modules.dlp.hot_path`;
- `modules.dlp.note`.
Если `modules.dlp.enabled=false`, Workforce UI должен продолжать работу и
показывать DLP/Security/Forensics как disabled или not configured, не превращая
это в ошибку основного рабочего экрана.
`GET /api/operator` также публикует additive runtime-state поля для первичного
экрана:
- `cache_status`;
- `modules.dlp.enabled`;
- `modules.dlp.status`;
- `modules.dlp.hot_path`;
- `modules.dlp.note`;
- `summary.severity`;
- `summary.blocks`.
Если `cache_status=warming`, клиент должен показать bounded stale/warming
state и не держать бесконечный loading indicator. Если
`modules.dlp.enabled=false`, operator screen должен считать DLP disabled-state
допустимым состоянием, а не ошибкой Workforce core.
## Что не меняется
- HTML-портал не удаляется.
+8
View File
@@ -50,6 +50,13 @@ Security layer:
Это не полноценная SIEM и не сертифицированная DLP.
Heavy DLP processing is not part of the required Workforce hot path. DLP
endpoint signals, screenshots, evidence review, heavy correlation and forensics
enrichment are treated as optional/deployment-specific modules. If the DLP
module is disabled or not configured, core Workforce reports and portal
readiness must remain available and the Security/Forensics views must show an
honest disabled/not configured state.
## Forensics Core
Forensics layer:
@@ -68,6 +75,7 @@ Optional addons and deployment-specific directions:
- pfSense;
- 1C;
- DLP endpoint signals and evidence workflow;
- AD/LDAP;
- SIEM/syslog;
- external storage;
-239
View File
@@ -1,239 +0,0 @@
# Workforce Operations Model
Статус: implemented in Worktime API and DetMir portal.
Модель отвечает на главный управленческий вопрос AWatch-rus Workforce:
рабочая активность сотрудников, загрузка, простои, перегруз и дисциплина
рабочего процесса. Это rule-based слой операционного контроля. Он не является
HR-оценкой, не использует ML/LLM и не выполняет автоматических санкций.
## Где смотреть
Основные точки:
- Worktime API:
`GET /reports/worktime/management?format=json`;
- Worktime HTML:
`GET /reports/worktime/management?format=html`;
- DetMir portal:
`/api/reports`, блок `workforce_operations`;
- UI портала:
роли `Руководитель` и вкладка `Отчеты`, блок `Операционная загрузка`.
## Источники
Модель использует только подтвержденные рабочие источники:
- ActivityWatch worktime rows;
- bucket рабочих сессий RDP;
- интервалы активности в рабочем окне;
- configured owner/department aliases;
- freshness/coverage metadata, которые уже возвращает Worktime API.
Отсутствие данных не считается простоем. При пропусках источников строка
получает `data_confidence=low` и guardrail
`low_confidence_not_for_discipline`.
## Runtime-настройки
Основной файл политики:
- пример: `configs/worktime-interpretation-policy.example.json`;
- runtime: `/etc/activitywatch/worktime-interpretation-policy.json`;
- env path: `AW_WORKTIME_MANAGER_INTERPRETATION_POLICY`.
Поля policy:
| Поле | Смысл | Рекомендуемое значение |
| --- | --- | --- |
| `underload_threshold` | порог недогруза от рабочего окна | `0.35..0.45` |
| `overload_threshold` | порог перегруза от рабочего окна | `1.10..1.25` |
| `drop_threshold_pct` | порог просадки тренда | `10..25` |
| `night_work_after` | начало вечернего/ночного отклонения | `20:00` |
| `weekend_work` | учитывать выходные отклонения | `true` |
| `min_trend_points` | минимум daily points для тренда | `3..7` |
| `off_hours_threshold_seconds` | минимум внерабочей активности для флага | `1800` |
`underload_threshold` и `overload_threshold` можно задавать дробью или
процентом: `0.45` равно `45`, `1.15` равно `115`.
Для перегруза effective threshold fail-closed зажат в диапазон `100..300`, чтобы
значение ниже 100% не создавало ложный статус перегруза.
Env fallback:
- `AW_WORKTIME_MANAGER_TARGET_COVERAGE_PCT`;
- `AW_WORKTIME_MANAGER_LOW_COVERAGE_PCT`;
- `AW_WORKTIME_MANAGER_OVERLOAD_COVERAGE_PCT`;
- `AW_WORKTIME_MANAGER_TREND_MIN_POINTS`;
- `AW_WORKTIME_MANAGER_TREND_DELTA_PCT`;
- `AW_WORKTIME_MANAGER_OFF_HOURS_THRESHOLD_SECONDS`;
- `AW_WORKTIME_MANAGER_NIGHT_WORK_AFTER`;
- `AW_WORKTIME_MANAGER_WEEKEND_WORK_ENABLED`.
Веса приложений остаются отдельной политикой:
- пример: `configs/detmir-workforce-policy.example.json`;
- runtime: `/etc/detmir-portal-workforce-policy.json`.
Она влияет на explainable KPI и weighted activity, но не подменяет
операционные статусы загрузки/простоя.
## API contract
`/reports/worktime/management?format=json` содержит:
```json
{
"workday": {
"target_coverage_pct": 75,
"low_coverage_pct": 35,
"overload_coverage_pct": 115
},
"workforce_operations": {
"status": "ATTENTION",
"summary": {},
"rows": [],
"model": {
"type": "rule_based",
"ml": false,
"llm": false,
"version": "workforce-operations-v1"
}
}
}
```
Каждая строка сотрудника содержит:
- `workday_active_seconds`, `workday_active_hhmm`;
- `workday_idle_seconds`, `workday_idle_hhmm`;
- `coverage_pct`;
- `load_status`;
- `idle_status`;
- `discipline_status`;
- `data_confidence`;
- `recommended_action`.
Полный roster в `rows[]` дополнительно содержит `operations`,
`operations.evidence`, `operations.guardrail` и
`operations_recommended_action`.
## Статусы загрузки
| Status | Значение | Действие |
| --- | --- | --- |
| `insufficient_data` | рабочее окно еще не началось или равно нулю | не делать вывод |
| `no_data` | нет сессий или worktime samples | проверить источники |
| `no_activity` | сессия/данные есть, активности в окне нет | проверить присутствие и задачи |
| `underloaded` | ниже low threshold | проверить загрузку и доступ к процессам |
| `below_target` | ниже target threshold | уточнить причину отклонения |
| `normal` | в рабочем диапазоне | наблюдать |
| `overloaded` | выше overload threshold | проверить переработку и риск аврала |
## Статусы простоя
| Status | Значение |
| --- | --- |
| `not_applicable` | нет рабочего окна |
| `unknown` | нет достаточных источников |
| `full_workday_idle_or_absent` | активность в рабочем окне отсутствует |
| `idle_detected` | простой выше порога |
| `no_significant_idle` | существенный простой не найден |
## Дисциплина процесса
`discipline_status` показывает отклонение от рабочего процесса, а не
автоматическое нарушение:
- `ok`;
- `off_hours`;
- `late_start`;
- `early_finish`;
- `multiple_flags`.
Для текущего дня `early_finish` не выставляется до завершения рабочего окна.
## Достоверность
`data_confidence`:
- `high`: есть session samples, worktime samples и active samples;
- `medium`: данных мало или нет active samples;
- `low`: нет сессий/worktime samples или рабочее окно невалидно.
Правило: low confidence строки сначала проверяются как проблема источников.
Их нельзя использовать как персональный дисциплинарный вывод.
## Summary
`workforce_operations.summary` содержит:
- `users_count`;
- `action_required_users`;
- `load.unknown_or_no_data_users`;
- `load.underloaded_users`;
- `load.normal_users`;
- `load.overloaded_users`;
- `idle.idle_users`;
- `discipline.review_users`;
- `confidence.low_users`;
- `confidence.medium_users`;
- `confidence.high_users`;
- `guardrail`.
Summary status:
- `LOW_CONFIDENCE`: нет строк или все строки low confidence;
- `ATTENTION`: есть перегруз, простой или дисциплинарные флаги;
- `WATCH`: есть недогруз, нет данных или low confidence;
- `OK`: отклонений нет.
## UI contract
Портал показывает отдельный блок `Операционная загрузка`:
- сводка: требуют разбора, недогруз, перегруз, простой, дисциплина, low
confidence;
- таблица сотрудников: active/idle/coverage/load/idle/discipline/confidence;
- рекомендуемое действие;
- guardrail и версию rule-based модели.
Это отдельный блок от `Почему такой индекс активности?`: explainable KPI
отвечает на вопрос "почему такой процент", а Workforce Operations отвечает
"кого и почему нужно разобрать".
## Ограничения
- Не утверждать автоматическую оценку эффективности сотрудника.
- Не считать missing data простоем.
- Не смешивать Security/Forensics claims с Workforce Operations.
- Не заявлять ML/LLM detection.
- Не выполнять автоматическое remediation/action.
- Не использовать GitHub Actions или демо-данные как registry release evidence.
## Проверка после изменения
Минимальный локальный контур:
```bash
cd /mnt/usb_hdd2/Projects/ActivityWatch-Russian/adk-rust
export CARGO_TARGET_DIR=/home/igor/.cache/detmir-adk-rust-target
cargo fmt --all --check
cargo test -p worktime-api -p detmir-portal --locked
cargo clippy -p worktime-api -p detmir-portal --all-targets --locked -- -D warnings
```
Минимальный live smoke:
```bash
curl -fsS 'http://10.10.10.13:5610/reports/worktime/management?format=json' \
| jq '.workforce_operations.summary'
curl -fsS 'http://10.10.10.2:8720/api/reports?role=manager' \
| jq '{status: .workforce_operations.summary.status, rows: (.workforce_operations.rows | length)}'
```
Браузерный smoke: открыть `http://10.10.10.2:8720/`, выбрать представление
менеджера и проверить блок `Операционная загрузка`. В рабочем состоянии должны
быть видны summary-карточки, таблица сотрудников, `workforce-operations-v1` и
guardrail про `low confidence`.
@@ -12,7 +12,6 @@ Environment=DETMIR_DLP_COMMAND=detmir-dlp
Environment=CONTOUR_CHECK_OUTPUT_ROOT=/var/lib/detmir-ai/contour-check-runs
Environment=CONTOUR_CHECK_ENV_FILE=/etc/detmir/detmir-check.env
Environment=DETMIR_PORTAL_URL=http://127.0.0.1:8720
Environment=DETMIR_DLP_ENABLED=false
EnvironmentFile=-/etc/detmir/detmir-check.env
EnvironmentFile=-/etc/awatch-rus/contour-check.env
ExecStart=/usr/bin/env bash /usr/local/sbin/awatch-contour-check
@@ -13,7 +13,6 @@ Environment=DETMIR_DLP_COMMAND=detmir-dlp
Environment=CONTOUR_CHECK_OUTPUT_ROOT=/var/lib/detmir-ai/contour-check-runs
Environment=CONTOUR_CHECK_ENV_FILE=/etc/detmir/detmir-check.env
Environment=DETMIR_PORTAL_URL=http://127.0.0.1:8720
Environment=DETMIR_DLP_ENABLED=false
EnvironmentFile=-/etc/detmir/detmir-check.env
EnvironmentFile=-/etc/awatch-rus/contour-check.env
ExecStart=/usr/bin/env bash /usr/local/sbin/awatch-contour-check
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=DetMir portal report cache prewarm
After=network-online.target detmir-portal.service
Wants=network-online.target detmir-portal.service
[Service]
Type=oneshot
User=igor
Group=igor
Environment=no_proxy=localhost,127.0.0.1,10.10.10.2
Environment=NO_PROXY=localhost,127.0.0.1,10.10.10.2
ExecStart=/usr/bin/curl -fsS --max-time 180 http://127.0.0.1:8720/api/reports -o /dev/null
TimeoutStartSec=210
Nice=19
IOSchedulingClass=idle
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Pre-warm DetMir portal report cache every 30 minutes
[Timer]
OnBootSec=5min
OnUnitActiveSec=30min
AccuracySec=1min
Persistent=true
Unit=detmir-portal-prewarm.service
[Install]
WantedBy=timers.target
@@ -0,0 +1,2 @@
[Service]
ExecStartPost=/bin/systemctl --no-block start detmir-portal-prewarm.service
-471
View File
@@ -1,471 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
MODE="repo"
AW_API="${AW_API:-http://127.0.0.1:5600}"
AW_WORKTIME_API="${AW_WORKTIME_API:-http://127.0.0.1:5610}"
AW_LOGICAL_HOST_ID="${AW_LOGICAL_HOST_ID:-${AW_MONITORED_WINDOWS_HOSTNAME:-SHARKON2025}}"
AW_READINESS_TIMEOUT_SECONDS="${AW_READINESS_TIMEOUT_SECONDS:-60}"
AW_READINESS_INTERVAL_SECONDS="${AW_READINESS_INTERVAL_SECONDS:-2}"
AW_EVENTS_LIMIT="${AW_EVENTS_LIMIT:-100}"
AW_EVENTS_MAX_SECONDS="${AW_EVENTS_MAX_SECONDS:-15}"
HAYA_ROOT="${AW_HAYABUSA_ROOT:-/opt/hayabusa}"
HAYA_DROP_DIR="${AW_HAYABUSA_DROP_DIR:-/opt/activitywatch/aw-rus-ops/drop}"
SQLITE_DB="${AW_SQLITE_DB:-/var/lib/activitywatch/aw-server-rust/sqlite.db}"
MAX_INCOMING_AGE_SECONDS="${MAX_HAYABUSA_INCOMING_AGE_SECONDS:-900}"
STRICT_SECRETS="${DETMIR_RESILIENCE_STRICT_SECRETS:-0}"
EXPECT_DLP_PROFILE="${DETMIR_RESILIENCE_EXPECT_DLP_PROFILE:-light}"
EXPECT_OPTIONAL_DLP_OFF="${DETMIR_RESILIENCE_EXPECT_OPTIONAL_DLP_OFF:-0}"
EXPECT_LOKI_OFF="${DETMIR_RESILIENCE_EXPECT_LOKI_OFF:-1}"
DLP_RUNTIME_UNITS=(
aw-dlp-influx-exporter.timer
aw-dlp-influx-exporter.service
activitywatch-dlp-aggregator.timer
activitywatch-dlp-aggregator.service
aw-dlp-report-scheduler.timer
aw-dlp-report-scheduler.service
aw-dlp-syslog-forwarder.timer
aw-dlp-syslog-forwarder.service
aw-dlp-webhook-sender.timer
aw-dlp-webhook-sender.service
aw-dlp-cef-exporter.timer
aw-dlp-cef-exporter.service
aw-dlp-ioc-refresh.timer
aw-dlp-ioc-refresh.service
aw-dlp-policy-engine.service
aw-dlp-case-management.service
detmir-portal-evidence.service
)
DLP_LIGHT_ALLOWED_UNITS=(
activitywatch-dlp-aggregator.timer
activitywatch-dlp-aggregator.service
aw-dlp-ioc-refresh.timer
aw-dlp-ioc-refresh.service
detmir-dlp-load-guard.timer
detmir-dlp-load-guard.service
)
DLP_HEAVY_RUNTIME_UNITS=(
aw-dlp-influx-exporter.timer
aw-dlp-influx-exporter.service
aw-dlp-report-scheduler.timer
aw-dlp-report-scheduler.service
aw-dlp-syslog-forwarder.timer
aw-dlp-syslog-forwarder.service
aw-dlp-webhook-sender.timer
aw-dlp-webhook-sender.service
aw-dlp-cef-exporter.timer
aw-dlp-cef-exporter.service
aw-dlp-policy-engine.service
aw-dlp-case-management.service
detmir-portal-evidence.service
)
LOKI_RUNTIME_UNITS=(
loki.service
promtail.service
)
OK_COUNT=0
WARN_COUNT=0
FAIL_COUNT=0
usage() {
cat <<'EOF'
Usage:
scripts/detmir_resilience_check.sh [--repo|--live|--all]
Modes:
--repo check repository hardening and docs only (default, CI-safe)
--live read-only checks for the local AW server/Hayabusa host
--all repo + live
Environment:
AW_API=http://127.0.0.1:5600
AW_WORKTIME_API=http://127.0.0.1:5610
AW_LOGICAL_HOST_ID=SHARKON2025
AW_READINESS_TIMEOUT_SECONDS=60
AW_EVENTS_MAX_SECONDS=15
AW_HAYABUSA_ROOT=/opt/hayabusa
AW_HAYABUSA_DROP_DIR=/opt/activitywatch/aw-rus-ops/drop
AW_SQLITE_DB=/var/lib/activitywatch/aw-server-rust/sqlite.db
DETMIR_RESILIENCE_EXPECT_DLP_PROFILE=light
DETMIR_RESILIENCE_EXPECT_OPTIONAL_DLP_OFF=0
DETMIR_RESILIENCE_EXPECT_LOKI_OFF=1
DETMIR_RESILIENCE_STRICT_SECRETS=1
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--repo) MODE="repo"; shift ;;
--live) MODE="live"; shift ;;
--all) MODE="all"; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
ok() {
OK_COUNT=$((OK_COUNT + 1))
printf '[OK] %s\n' "$*"
}
warn() {
WARN_COUNT=$((WARN_COUNT + 1))
printf '[WARN] %s\n' "$*"
}
fail() {
FAIL_COUNT=$((FAIL_COUNT + 1))
printf '[FAIL] %s\n' "$*"
}
have() {
command -v "$1" >/dev/null 2>&1
}
require_file() {
local path="$1"
if [[ -f "$ROOT_DIR/$path" ]]; then
ok "file exists: $path"
else
fail "missing file: $path"
fi
}
require_pattern() {
local path="$1"
local pattern="$2"
local label="$3"
if grep -Eq "$pattern" "$ROOT_DIR/$path"; then
ok "$label"
else
fail "$label"
fi
}
check_repo() {
printf '== repo resilience checks ==\n'
require_file "scripts/detmir_resilience_check.sh"
require_file "docs/DETMIR_RESILIENCE_HARDENING_RU.md"
require_file "docs/DLP_RESOURCE_PROFILES_RU.md"
require_file "docs/DLP_OPTIONAL_RUNTIME_RU.md"
require_file "scripts/detmir_dlp_load_guard.sh"
require_file "scripts/detmir_dlp_warehouse_sync.sh"
require_file "aw-server/hayabusa/aw-hayabusa.sh"
require_file "adk-rust/crates/hayabusa-tools/src/bin/autoprocess.rs"
require_file "windows/AWatchRusCollectorGuardService.cs"
require_file "windows/install-collector-guard-service.ps1"
require_pattern "aw-server/hayabusa/aw-hayabusa.sh" "HAYA_QUARANTINE_DIR" "Hayabusa wrapper has quarantine root"
require_pattern "aw-server/hayabusa/aw-hayabusa.sh" "quarantine_incoming_package" "Hayabusa wrapper isolates incoming poison packages"
require_pattern "adk-rust/crates/hayabusa-tools/src/bin/autoprocess.rs" "validate_drop_inputs" "Hayabusa autoprocess validates drop package before accept"
require_pattern "adk-rust/crates/hayabusa-tools/src/bin/autoprocess.rs" "quarantine_drop_package" "Hayabusa autoprocess quarantines bad drop package"
require_pattern "windows/AWatchRusCollectorGuardService.cs" "Process\\.Exited|ChildExited" "Collector guard service watches child exit"
require_pattern "windows/AWatchRusCollectorGuardService.cs" "MaxChildRestartsInWindow" "Collector guard service has bounded child restart budget"
require_pattern "windows/install-collector-guard-service.ps1" "failureflag" "Collector guard installer enables SCM failureflag"
require_pattern "docs/DETMIR_RESILIENCE_HARDENING_RU.md" "Hayabusa poison-package isolation" "Resilience doc records Hayabusa hardening"
require_pattern "docs/DETMIR_RESILIENCE_HARDENING_RU.md" "Windows collector guard service child watchdog" "Resilience doc records guard child watchdog"
require_pattern "docs/DLP_RESOURCE_PROFILES_RU.md" "core_only" "DLP resource profiles document core_only"
require_pattern "docs/DLP_RESOURCE_PROFILES_RU.md" "auto.?disable|автоотключ" "DLP resource profiles document auto-disable guard"
require_pattern "docs/DLP_RESOURCE_PROFILES_RU.md" "rollback" "DLP resource profiles document rollback"
require_pattern "docs/DETMIR_CURRENT_STATE_RU.md" "AW_DLP_PROFILE=light" "Current state records DLP light profile"
require_pattern "scripts/detmir_dlp_runtime_control.sh" "set-profile" "DLP runtime control supports profile switching"
require_pattern "scripts/detmir_dlp_runtime_control.sh" "rollback_dlp" "DLP runtime control supports rollback"
require_pattern "scripts/detmir_dlp_load_guard.sh" "set-profile core_only" "DLP load guard can auto-disable DLP to core_only"
require_pattern "scripts/detmir_dlp_load_guard.sh" "STRIKES_REQUIRED" "DLP load guard requires consecutive overload checks"
require_pattern "scripts/detmir_dlp_warehouse_sync.sh" "sqlite3 .*\\.backup" "DLP warehouse sync uses SQLite backup"
require_pattern "ansible/group_vars/all.yml" 'aw_dlp_profile: "light"' "Production defaults keep DLP profile light"
require_pattern "ansible/group_vars/all.yml" 'aw_dlp_enabled: true' "Production defaults enable lightweight DLP"
require_pattern "ansible/group_vars/all.yml" 'aw_dlp_influx_enabled: false' "Production defaults keep DLP Influx disabled"
require_pattern "ansible/group_vars/all.yml" 'aw_dlp_light_collector_enabled: true' "Production defaults enable lightweight DLP collector"
require_pattern "ansible/group_vars/all.yml" 'aw_dlp_light_guard_enabled: true' "Production defaults enable DLP load guard"
require_pattern "ansible/group_vars/all.yml" 'detmir_portal_dlp_module_enabled_override: true' "Production defaults expose DLP light status to portal"
require_pattern "ansible/deploy_aw_server.yml" "detmir-dlp-load-guard.service" "AW server deploy installs DLP load guard service"
require_pattern "ansible/deploy_aw_server.yml" "CPUQuota=.*aw_dlp_aggregator_cpu_quota" "DLP aggregator has systemd CPU quota"
require_pattern "ansible/deploy_detmir_portal.yml" "detmir-dlp-warehouse-sync.service" "Portal deploy installs DLP warehouse sync service"
require_pattern "ansible/deploy_detmir_portal.yml" "detmir_portal_dlp_module_enabled_override \\| default\\(false\\)" "Portal deploy defaults DLP module to disabled"
if [[ -f "$ROOT_DIR/ansible/inventory.ini" ]] && grep -Eq '(^|[[:space:]])ansible_(become_)?password[[:space:]]*=[[:space:]]*[^<{]' "$ROOT_DIR/ansible/inventory.ini"; then
if [[ "$STRICT_SECRETS" == "1" ]]; then
fail "ansible/inventory.ini appears to contain literal password assignments; move them to vault/env"
else
warn "ansible/inventory.ini appears to contain literal password assignments; strict mode would fail"
fi
else
ok "no literal ansible password assignments detected in ansible/inventory.ini"
fi
}
check_systemd_unit() {
local unit="$1"
if ! have systemctl; then
warn "systemctl unavailable; skipping $unit"
return
fi
if systemctl is-active --quiet "$unit"; then
ok "systemd active: $unit"
else
fail "systemd not active: $unit"
fi
}
check_aw_api() {
if ! have curl; then
warn "curl unavailable; skipping AW API check"
return
fi
local deadline last_code elapsed
deadline=$((SECONDS + AW_READINESS_TIMEOUT_SECONDS))
last_code=""
while (( SECONDS <= deadline )); do
elapsed="$(
curl -sS --connect-timeout 3 --max-time 8 -o /dev/null -w '%{http_code} %{time_total}' \
"$AW_API/api/0/info" 2>/dev/null || true
)"
last_code="${elapsed%% *}"
if [[ "$last_code" == "200" ]]; then
ok "ActivityWatch API readiness /api/0/info returns 200 (${elapsed#* }s)"
return
fi
sleep "$AW_READINESS_INTERVAL_SECONDS"
done
case "$last_code" in
503) fail "ActivityWatch API readiness ended on 503; possible datastore lock poisoning" ;;
""|000) fail "ActivityWatch API did not become ready within ${AW_READINESS_TIMEOUT_SECONDS}s" ;;
*) fail "ActivityWatch API readiness unexpected final HTTP status: $last_code" ;;
esac
}
check_aw_hot_path() {
if ! have curl; then
warn "curl unavailable; skipping AW hot-path event check"
return
fi
local url result code elapsed
url="${AW_API%/}/api/0/buckets/aw-worktime-sessions_${AW_LOGICAL_HOST_ID}/events?limit=${AW_EVENTS_LIMIT}"
result="$(curl -sS --connect-timeout 3 --max-time "$AW_EVENTS_MAX_SECONDS" -o /dev/null -w '%{http_code} %{time_total}' "$url" 2>/dev/null || true)"
code="${result%% *}"
elapsed="${result#* }"
if [[ "$code" != "200" ]]; then
fail "ActivityWatch worktime events hot path returned HTTP ${code:-none}"
return
fi
if awk -v t="$elapsed" -v max="$AW_EVENTS_MAX_SECONDS" 'BEGIN { exit !(t <= max) }'; then
ok "ActivityWatch worktime events hot path returns 200 (${elapsed}s, limit=${AW_EVENTS_LIMIT})"
else
fail "ActivityWatch worktime events hot path exceeded ${AW_EVENTS_MAX_SECONDS}s (${elapsed}s)"
fi
}
check_worktime_api() {
if ! have curl; then
warn "curl unavailable; skipping Worktime API check"
return
fi
local tmp code
tmp="$(mktemp)"
code="$(curl -sS --connect-timeout 3 --max-time 12 -o "$tmp" -w '%{http_code}' \
"${AW_WORKTIME_API%/}/reports/worktime/today?format=json&host=${AW_LOGICAL_HOST_ID}&allow_stale=1" 2>/dev/null || true)"
if [[ "$code" != "200" ]]; then
fail "Worktime API today report returned HTTP ${code:-none}"
rm -f "$tmp"
return
fi
if have jq; then
if jq -e '(.degraded // false) == false' "$tmp" >/dev/null 2>&1; then
ok "Worktime API today report is not degraded"
else
fail "Worktime API today report is degraded"
fi
if jq -e '((.rows // .users // []) | length) > 0' "$tmp" >/dev/null 2>&1; then
ok "Worktime API today report has employee rows"
else
fail "Worktime API today report has no employee rows"
fi
else
ok "Worktime API today report returns 200"
fi
rm -f "$tmp"
}
check_failed_units() {
if ! have systemctl; then
return
fi
local failed
failed="$(systemctl --failed --no-legend --plain 2>/dev/null | wc -l | tr -d ' ')"
if [[ "$failed" == "0" ]]; then
ok "systemd failed units count is 0"
else
fail "systemd failed units count is $failed"
fi
}
count_files() {
local dir="$1"
local pattern="$2"
if [[ ! -d "$dir" ]]; then
printf '0\n'
return
fi
find "$dir" -maxdepth 1 -type f -name "$pattern" | wc -l | tr -d ' '
}
check_hayabusa_queues() {
local incoming_dir="$HAYA_ROOT/inbox/incoming"
local quarantine_dir="$HAYA_ROOT/quarantine"
local incoming_count drop_count old_count quarantine_count
incoming_count="$(count_files "$incoming_dir" '*.zip')"
drop_count="$(count_files "$HAYA_DROP_DIR" '*.zip')"
old_count="0"
if [[ -d "$incoming_dir" ]]; then
old_count="$(find "$incoming_dir" -maxdepth 1 -type f -name '*.zip' -mmin "+$((MAX_INCOMING_AGE_SECONDS / 60))" | wc -l | tr -d ' ')"
fi
if [[ "$incoming_count" == "0" ]]; then
ok "Hayabusa incoming zip count is 0"
else
warn "Hayabusa incoming zip count is $incoming_count"
fi
if [[ "$drop_count" == "0" ]]; then
ok "Hayabusa drop zip count is 0"
else
warn "Hayabusa drop zip count is $drop_count"
fi
if [[ "$old_count" == "0" ]]; then
ok "Hayabusa incoming has no stale zip older than ${MAX_INCOMING_AGE_SECONDS}s"
else
fail "Hayabusa incoming has $old_count stale zip package(s)"
fi
if [[ -d "$quarantine_dir" ]]; then
quarantine_count="$(find "$quarantine_dir" -type f -name reason.json | wc -l | tr -d ' ')"
if [[ "$quarantine_count" == "0" ]]; then
ok "Hayabusa quarantine reason count is 0"
else
warn "Hayabusa quarantine reason count is $quarantine_count; review/replay policy required"
fi
else
warn "Hayabusa quarantine root not found yet: $quarantine_dir"
fi
}
check_sqlite_files() {
if [[ ! -f "$SQLITE_DB" ]]; then
warn "AW SQLite DB not found at $SQLITE_DB; skipping local DB size check"
return
fi
local db_size wal_size
db_size="$(stat -c '%s' "$SQLITE_DB")"
wal_size="0"
[[ -f "$SQLITE_DB-wal" ]] && wal_size="$(stat -c '%s' "$SQLITE_DB-wal")"
if (( db_size > 5 * 1024 * 1024 * 1024 )); then
fail "AW SQLite DB exceeds 5GiB"
elif (( db_size > 2 * 1024 * 1024 * 1024 )); then
warn "AW SQLite DB exceeds 2GiB"
else
ok "AW SQLite DB size below 2GiB"
fi
if (( wal_size > 1024 * 1024 * 1024 )); then
fail "AW SQLite WAL exceeds 1GiB"
elif (( wal_size > 256 * 1024 * 1024 )); then
warn "AW SQLite WAL exceeds 256MiB"
else
ok "AW SQLite WAL size below 256MiB"
fi
}
check_sqlite_hot_path_index() {
if [[ ! -f "$SQLITE_DB" ]]; then
warn "AW SQLite DB not found at $SQLITE_DB; skipping hot-path index check"
return
fi
if ! have sqlite3; then
warn "sqlite3 unavailable; skipping hot-path index check"
return
fi
local index_exists plan
index_exists="$(sqlite3 "$SQLITE_DB" "SELECT name FROM sqlite_master WHERE type='index' AND name='events_bucketrow_starttime_desc_index';" 2>/dev/null || true)"
if [[ "$index_exists" == "events_bucketrow_starttime_desc_index" ]]; then
ok "AW SQLite hot-path index exists"
else
fail "AW SQLite hot-path index events_bucketrow_starttime_desc_index is missing"
return
fi
plan="$(
sqlite3 "$SQLITE_DB" "EXPLAIN QUERY PLAN SELECT id,starttime,endtime,data FROM events WHERE bucketrow=(SELECT id FROM buckets WHERE name='aw-worktime-sessions_${AW_LOGICAL_HOST_ID}') ORDER BY starttime DESC LIMIT ${AW_EVENTS_LIMIT};" 2>/dev/null || true
)"
if printf '%s' "$plan" | grep -q 'events_bucketrow_starttime_desc_index'; then
ok "AW SQLite worktime event query uses hot-path index"
else
fail "AW SQLite worktime event query does not use hot-path index"
fi
if printf '%s' "$plan" | grep -q 'TEMP B-TREE'; then
fail "AW SQLite worktime event query still builds TEMP B-TREE"
else
ok "AW SQLite worktime event query avoids TEMP B-TREE"
fi
}
check_units_inactive() {
local label="$1"
shift
if ! have systemctl; then
warn "systemctl unavailable; skipping $label runtime check"
return
fi
local active_units=()
local unit active
for unit in "$@"; do
active="$(systemctl is-active "$unit" 2>/dev/null || true)"
if [[ "$active" == "active" || "$active" == "activating" ]]; then
active_units+=("$unit:$active")
fi
done
if [[ "${#active_units[@]}" -eq 0 ]]; then
ok "$label runtime units are inactive"
else
fail "$label runtime units active: ${active_units[*]}"
fi
}
check_live() {
printf '== live resilience checks ==\n'
check_systemd_unit "activitywatch-server"
check_systemd_unit "aw-worktime-api"
check_aw_api
check_aw_hot_path
check_worktime_api
check_failed_units
check_hayabusa_queues
check_sqlite_files
check_sqlite_hot_path_index
if [[ "$EXPECT_OPTIONAL_DLP_OFF" == "1" || "$EXPECT_DLP_PROFILE" == "core_only" ]]; then
check_units_inactive "optional DLP" "${DLP_RUNTIME_UNITS[@]}"
elif [[ "$EXPECT_DLP_PROFILE" == "light" ]]; then
check_units_inactive "heavy DLP" "${DLP_HEAVY_RUNTIME_UNITS[@]}"
else
warn "optional DLP runtime inactive check skipped for profile=$EXPECT_DLP_PROFILE"
fi
if [[ "$EXPECT_LOKI_OFF" == "1" ]]; then
check_units_inactive "Loki" "${LOKI_RUNTIME_UNITS[@]}"
else
warn "Loki inactive check skipped by env"
fi
}
case "$MODE" in
repo) check_repo ;;
live) check_live ;;
all) check_repo; check_live ;;
*) fail "invalid mode: $MODE" ;;
esac
printf 'summary: ok=%s warn=%s fail=%s\n' "$OK_COUNT" "$WARN_COUNT" "$FAIL_COUNT"
if (( FAIL_COUNT > 0 )); then
exit 1
fi