AWatch-rus/RECOVERY_AUDIT.md

Operational Recovery Audit

Date: 2026-07-01

Scope: repository-first audit of the current AWatch-rus / DetMir operational recovery capability. This document describes only mechanisms already present in the repository. It does not define new backup or recovery mechanisms.

Evidence Sources

The audit reviewed these repository sources:

• docs/BACKUP_AND_RECOVERY_RU.md
• docs/RETENTION_POLICY_RU.md
• docs/OPERATIONS_RUNBOOK_RU.md
• docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md
• docs/DETMIR_RESTORE_BASELINE_2026-06-29_RU.md
• scripts/prod_backup_restore.sh
• adk-rust/crates/prod-backup-restore/src/main.rs
• ansible/deploy_aw_server.yml
• ansible/deploy_aw_windows.yml
• ansible/post_validate_aw_windows.yml
• aw-server/*.service, aw-server/*.timer, aw-server/*.path
• windows/ActivityWatch.Windows.Common.psm1
• windows/hardening-recovery.ps1
• windows/rebuild-worktime-tasks.ps1
• windows/fix-session-watchers.ps1
• windows/cleanup-disc-sessions.ps1
• clickhouse-1c/docker-compose.yml
• clickhouse-workforce/docker-compose.yml
• grafana-1c/docker-compose.yml
• docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md
• docs/registry/registry-evidence-manifest.json

Current Recovery Procedure Actually Supported

Server-side service recovery

The repository supports service restart and health validation through systemd, Ansible deployment, and smoke checks:

• activitywatch-server.service restarts on failure and runs with the configured AW server env file. Evidence: aw-server/activitywatch-server.service.
• ansible/deploy_aw_server.yml installs and restarts activitywatch-server.service, then waits for /api/0/info. Evidence: ansible/deploy_aw_server.yml:2365-2375.
• The generic operations runbook requires checking /healthz, /readyz, /metrics, service status, journal, and smoke scripts after recovery. Evidence: docs/OPERATIONS_RUNBOOK_RU.md:59-71.
• The production validation runbook defines the DetMir smoke sequence through check-aw-data.sh, check-aw-full.sh, and local contour smoke. Evidence: docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md.

ActivityWatch SQLite maintenance and recovery support

The repository supports guarded maintenance and limited DB merge/migration flows, not a complete automated restore:

• Weekly guarded SQLite maintenance is installed as aw-db-maintenance.service / aw-db-maintenance.timer. Evidence: aw-server/aw-db-maintenance.service, aw-server/aw-db-maintenance.timer.
• Optional SQLite vacuum is defined as aw-db-vacuum.service / aw-db-vacuum.timer. It is opt-in in Ansible through aw_db_vacuum_timer_enabled. Evidence: aw-server/aw-db-vacuum.service, aw-server/aw-db-vacuum.timer, ansible/deploy_aw_server.yml:613-649.
• Legacy root DB merge is implemented in ansible/deploy_aw_server.yml when aw_legacy_db_merge_enabled is true. It stops the service, backs up target and legacy DB files, merges them with merge-aw-server-dbs, installs the merged DB, restarts the service, and waits for API readiness. Evidence: ansible/deploy_aw_server.yml:2248-2375.
• A separate Rust prod-backup-restore binary exists only as a plan/checker. It explicitly rejects --apply. Evidence: adk-rust/crates/prod-backup-restore/src/main.rs:100-105.

Windows collector recovery

The Windows side has the strongest implemented recovery automation:

• Deployment writes deployment-config.json, launcher scripts, and recovery-loop.ps1. Evidence: windows/deploy-domain-users.ps1, windows/hardening-recovery.ps1.
• Write-ActivityWatchRecoveryScript generates a script that imports ActivityWatch.Windows.Common.psm1 and calls Invoke-ActivityWatchRecoveryLoop. Evidence: windows/ActivityWatch.Windows.Common.psm1:1576-1597.
• Register-ActivityWatchRecoveryTask creates the scheduled task ActivityWatch Recovery, using an interactive user when possible and SYSTEM fallback otherwise. Evidence: windows/ActivityWatch.Windows.Common.psm1:2447-2488.
• Invoke-ActivityWatchRecoveryLoop is an actual loop: it uses a lock file, cleans non-live session processes, starts the worktime session collector when allowed, starts configured live user launch tasks, and uses console fallback. Evidence: windows/ActivityWatch.Windows.Common.psm1:2114-2168.
• Start-ActivityWatchTasks starts launch tasks for live users and starts the recovery task. Evidence: windows/ActivityWatch.Windows.Common.psm1:2656-2672.
• rebuild-worktime-tasks.ps1, fix-session-watchers.ps1, and cleanup-disc-sessions.ps1 provide manual repair paths for task/script regeneration, stale recovery loop restart, and disconnected-session cleanup.

Hayabusa intake recovery

The repository supports event-driven reprocessing of uploaded Hayabusa packages:

• aw-hayabusa-drop.path watches /opt/activitywatch/aw-rus-ops/drop for zip packages and triggers aw-hayabusa-drop.service. Evidence: aw-server/aw-hayabusa-drop.path.
• aw-hayabusa-drop.service runs /usr/local/bin/aw-hayabusa-autoprocess. Evidence: aw-server/aw-hayabusa-drop.service.
• ansible/deploy_aw_server.yml creates Hayabusa directories, installs pinned Hayabusa, installs Rust helpers, creates the drop zone, enables aw-hayabusa-drop.path, and runs aw-hayabusa doctor. Evidence: ansible/deploy_aw_server.yml:2780-3070.

Readiness evidence recovery support

The repository supports periodic readiness bundle generation:

• detmir-readiness.service writes readiness output to /var/lib/activitywatch/health/readiness-bundle. Evidence: aw-server/detmir-readiness.service.
• detmir-readiness.timer runs daily with persistent timer behavior. Evidence: aw-server/detmir-readiness.timer.

Current Backup Procedure Actually Supported

ActivityWatch server local backup artifacts

Implemented:

• ansible/deploy_aw_server.yml creates /var/lib/activitywatch/backups and /var/lib/activitywatch/backups/db. Evidence: ansible/deploy_aw_server.yml:80-128, ansible/deploy_aw_server.yml:2263-2269.
• Before legacy DB merge, Ansible copies the target DB and legacy root DB into timestamped files under {{ aw_server_data_dir }}/backups/db. Evidence: ansible/deploy_aw_server.yml:2274-2294.
• Before applying server-side settings/views/classes, Ansible copies current payloads to timestamped JSON backups under {{ aw_server_data_dir }}/backups. Evidence: ansible/deploy_aw_server.yml:2406-2420.
• aw-prune-local-state.timer prunes old local backups using configured retention and keep-last values. Evidence: ansible/deploy_aw_server.yml:236-281, docs/RETENTION_POLICY_RU.md:70-71.

Not currently implemented:

• A general scheduled full backup of the active ActivityWatch SQLite DB.
• A repository-defined off-host backup copy for AW DB, configs, Grafana, ClickHouse, DLP evidence, Hayabusa archives, or Windows collector state.

Windows package rollback backup

Implemented:

• Install-ActivityWatchPackage backs up the existing install root into install-<timestamp> before replacing it, keeps only the latest two install backups, checks free space, and cleans temporary extraction directories. Evidence: windows/ActivityWatch.Windows.Common.psm1:80-95, windows/ActivityWatch.Windows.Common.psm1:148-245.
• hardening-recovery.ps1 can run Install-ActivityWatchPackage when -RepairPackage is supplied. Evidence: windows/hardening-recovery.ps1.

Not currently implemented:

• Automatic restore from the saved Windows install-* backup directory.
• Backup of C:\ProgramData\AWatch-rus as a whole before repair.

Gitea registry backup

Implemented as registry-readiness support, not DetMir runtime recovery:

• Registry docs define Gitea backup path, script, systemd service/timer, gitea dump ZIP format, SHA256 checksum, daily schedule, and 14-day retention. Evidence: docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:7-23, docs/registry/registry-evidence-manifest.json:26-39.

Not currently implemented:

• Tested Gitea restore. The manifest explicitly says restore_tested=false and production_ready=false. Evidence: docs/registry/registry-evidence-manifest.json:37-39.

Current Restore Procedure

ActivityWatch DB merge/restore-like flow

Supported:

1. Enable aw_legacy_db_merge_enabled.
2. Deploy server playbook.
3. Playbook checks legacy root DB and target DB.
4. Playbook stops activitywatch-server.service.
5. Playbook backs up target and legacy DB files.
6. Playbook runs /usr/local/bin/merge-aw-server-dbs.
7. Playbook installs the merged DB as active target DB.
8. Playbook restarts activitywatch-server.service.
9. Playbook waits for /api/0/info.

Evidence: ansible/deploy_aw_server.yml:2248-2375.

Not currently implemented:

• A generic "restore selected backup file to active AW DB" command.
• A tested end-to-end AW DB restore runbook.
• Automated checksum verification for AW DB backup files.
• Automated rollback from a failed DB merge to the backup file.

prod-backup-restore plan-only flow

Supported:

1. scripts/prod_backup_restore.sh locates the Rust planner binary or exits with build instructions. Evidence: scripts/prod_backup_restore.sh:14-29.
2. The Rust planner reads private-config/runtime.env if available, checks required env vars, checks sshpass, ansible-playbook, inventory, and merge-aw-server-dbs. Evidence: adk-rust/crates/prod-backup-restore/src/main.rs:111-131, adk-rust/crates/prod-backup-restore/src/main.rs:175-209.
3. The planner prints planned commands including remote backup directory creation, DB copies, service stop, merge, install, and Ansible validation. Evidence: adk-rust/crates/prod-backup-restore/src/main.rs:211-280.

Not currently implemented:

• Execution of the planned restore. --apply fails by design. Evidence: adk-rust/crates/prod-backup-restore/src/main.rs:100-105.

Windows collector recovery

Supported:

1. Run ActivityWatch Recovery scheduled task or let it run on its configured trigger.
2. Recovery loop enforces a single lock.
3. Recovery loop stops collectors/watchers in non-live sessions.
4. Recovery loop starts the global worktime collector when allowed.
5. Recovery loop starts configured live-user launch tasks.
6. Recovery loop attempts console fallback when no configured live task starts.

Evidence: windows/ActivityWatch.Windows.Common.psm1:2114-2168, windows/ActivityWatch.Windows.Common.psm1:2447-2488, windows/ActivityWatch.Windows.Common.psm1:2656-2672.

Not currently implemented:

• Automatic reconstruction of lost Windows state from an external backup.
• Automatic restore of Windows collector queues after corruption or deletion.

Gitea restore

Supported:

• Manual outline exists: prepare isolated test server, install same Gitea version, stop Gitea, verify checksum, unpack dump, restore app/data/repos/db according to Gitea official procedure, fix ownership, start Gitea, run gitea doctor check, regenerate hooks if needed, and run post-restore checks. Evidence: docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:42-90.

Not currently implemented:

• Tested Gitea restore. Evidence: docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:92-100.

Components That Cannot Yet Be Restored Automatically

The following are confirmed by repository inspection:

Component	Current state	Evidence
ActivityWatch active SQLite DB	Backups exist around merge/settings operations, but generic restore is Not currently implemented.	ansible/deploy_aw_server.yml:2274-2294, adk-rust/crates/prod-backup-restore/src/main.rs:100-105
ClickHouse 1C data	Docker volume exists; no backup/restore automation found. Not currently implemented.	clickhouse-1c/docker-compose.yml:13-19
ClickHouse Workforce data	Docker volume exists; no backup/restore automation found. Not currently implemented.	clickhouse-workforce/docker-compose.yml:11-17
Grafana data volume	Docker volume exists; no repo cleanup and no restore automation. Not currently implemented.	grafana-1c/docker-compose.yml:67-79, docs/RETENTION_POLICY_RU.md:99
Prometheus TSDB	Retention configured, backup/restore not documented. Not currently implemented.	grafana-1c/docker-compose.yml:40-49
DLP policy/case/warehouse DBs	Retention doc says no automatic deletion; backup/restore not implemented. Not currently implemented.	docs/RETENTION_POLICY_RU.md:43-47, docs/RETENTION_POLICY_RU.md:87-90
DLP evidence and compliance reports	Cleanup disabled; restore depends on customer backup if manually deleted. Not currently implemented.	docs/RETENTION_POLICY_RU.md:221-223
Hayabusa reports/archive	Processing is automated; restore of archive/reports is not automated. Not currently implemented.	docs/RETENTION_POLICY_RU.md:48, docs/RETENTION_POLICY_RU.md:91
Windows collector state and queues	Recovery restarts collectors; external backup/restore of state is not implemented.	docs/RETENTION_POLICY_RU.md:40, docs/RETENTION_POLICY_RU.md:85
Diagnostic bundles and release evidence	No automatic backup/restore found. Not currently implemented.	docs/RETENTION_POLICY_RU.md:100-101

Components Requiring Manual Intervention

• AW DB merge/recovery: operator must enable aw_legacy_db_merge_enabled, run Ansible, review backup files, and verify API. Evidence: ansible/deploy_aw_server.yml:2248-2375.
• prod-backup-restore: operator can only review a plan; execution is manual because --apply is disabled. Evidence: adk-rust/crates/prod-backup-restore/src/main.rs:100-105.
• Windows collector recovery after severe state loss: operator must use hardening-recovery.ps1, rebuild-worktime-tasks.ps1, fix-session-watchers.ps1, or redeploy. Evidence: windows/*.ps1.
• Hayabusa stuck path/service: operator may need to reset failed units, repair drop-zone permissions, and rerun processing. Evidence: aw-server/aw-hayabusa-drop.path, aw-server/aw-hayabusa-drop.service.
• Gitea restore: manual isolated test restore is required; tested restore is not yet recorded. Evidence: docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:42-100.
• ClickHouse/Grafana/Prometheus/DLP data restore: Not currently implemented.

Missing Documentation

Critical and high-confidence gaps only:

• Exact AW DB restore runbook from /var/lib/activitywatch/backups/db to the active DB path. Not currently implemented.
• Post-restore verification checklist for AW DB backup restore, including checksum, ownership, service restart, bucket freshness, and worktime report checks. Not currently implemented.
• Backup inventory mapping each persistent component to an actual backup owner, schedule, storage location, retention, and restore command. Partially covered by docs/BACKUP_AND_RECOVERY_RU.md and docs/RETENTION_POLICY_RU.md, but operational restore ownership is Not currently implemented.
• ClickHouse 1C and ClickHouse Workforce backup/restore runbooks. Not currently implemented.
• Grafana volume restore runbook. Not currently implemented.
• DLP evidence/case/policy restore runbook. Not currently implemented.
• Windows C:\ProgramData\AWatch-rus state backup/restore runbook. Not currently implemented.
• Gitea restore evidence result. The runbook exists, but restore test is marked false. Evidence: docs/registry/registry-evidence-manifest.json:37-39.

Missing Automation

• Automated AW DB restore from a selected backup file. Not currently implemented.
• Automated AW DB backup with checksum on a schedule independent of merge operations. Not currently implemented.
• Off-host/offline copy for AW DB, ClickHouse volumes, Grafana data, DLP evidence, Hayabusa archives, Windows state, and release evidence. Not currently implemented.
• ClickHouse backup and restore automation. Not currently implemented.
• Grafana data volume backup and restore automation. Not currently implemented.
• DLP evidence/case/policy backup and restore automation. Not currently implemented.
• Windows state backup and restore automation. Not currently implemented.
• Automated restore drill evidence generation. Not currently implemented.

Operational Risks

Risk	Severity	Evidence	Impact
Restore is partially plan-only for AW DB	Critical	prod-backup-restore rejects --apply	Operator can plan but cannot run a deterministic automated restore through this tool
AW DB backups are created around specific operations, not as a general scheduled full backup	Critical	ansible/deploy_aw_server.yml:2274-2294	A recent recovery point may be unavailable if no merge/settings operation occurred
ClickHouse data has no repo-defined restore path	Critical	ClickHouse Docker volumes only	Loss/corruption of 1C or workforce ClickHouse data requires ad hoc operator recovery
DLP/Hayabusa evidence has no automated restore	High	docs/RETENTION_POLICY_RU.md:87-91, docs/RETENTION_POLICY_RU.md:221-223	Forensic/case continuity depends on external/customer backup
Grafana data volume has no repo-defined backup/restore	High	grafana-1c/docker-compose.yml:67-79	Dashboard DB/users/session state may require manual reconstruction even though provisioned dashboards exist
Gitea restore is documented but untested	High	restore_tested=false	Source-control recovery confidence remains limited
Windows recovery restarts collectors but does not restore deleted state	Medium	Windows recovery loop evidence	Collector state/queues/logs can be lost if state root is deleted
Generic backup document is intentionally high-level	Medium	docs/BACKUP_AND_RECOVERY_RU.md:5-6	Operators need component-specific procedures during incidents
Hayabusa path recovery depends on service/path health and permissions	Medium	aw-hayabusa-drop.path, aw-hayabusa-drop.service	Drop backlog or permission drift can stall forensic intake
Recovery smoke is documented but not tied to a single recovery command	Low	docs/OPERATIONS_RUNBOOK_RU.md, docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md	Operator can validate, but command sequencing remains manual

Recovery Confidence Score

Overall score: 48 / 100

Justification:

• Windows collector process recovery is mature: scheduled task, lock, live session handling, cleanup of non-live sessions, and task restart are implemented.
• Server service recovery and smoke validation are present.
• Local maintenance and retention are present.
• AW DB merge has safety backups and API validation, but generic restore is not implemented.
• prod-backup-restore is explicitly plan-only.
• ClickHouse, Grafana, Prometheus, DLP evidence/cases, Hayabusa archive, Windows state, diagnostic bundles, and release evidence do not have automated restore procedures in the repository.
• Gitea backup is documented with checksum and timer metadata, but restore is marked untested.

Prioritized Confirmed Gaps

Critical

1. Generic AW DB restore from backup is Not currently implemented. Evidence: backup files are created by Ansible, but prod-backup-restore refuses --apply.

2. Scheduled full AW DB backup independent of merge/settings changes is Not currently implemented. Evidence: backups are tied to merge/settings operations and pruning exists, but no scheduled full DB backup unit is present.

3. ClickHouse 1C and Workforce backup/restore are Not currently implemented. Evidence: both stacks persist to Docker volumes; no backup/restore runbook or automation is present.

High

4. DLP evidence/case/policy restore is Not currently implemented. Evidence: retention policy explicitly leaves DLP evidence/cases without automatic cleanup and says recovery depends on customer backup after manual deletion.

5. Hayabusa reports/archive restore is Not currently implemented. Evidence: intake processing is automated, but archive/report restore is not.

6. Grafana data volume backup/restore is Not currently implemented. Evidence: Grafana uses grafana-data; no repo restore procedure exists.

7. Gitea restore test is Not currently implemented. Evidence: registry manifest has restore_tested=false.

Medium

8. Windows state root backup/restore is Not currently implemented. Evidence: Windows recovery restarts collectors and repairs tasks, but does not restore C:\ProgramData\AWatch-rus from backup.

9. Component-specific recovery ownership matrix is Not currently implemented. Evidence: generic backup doc says component list must be refined by release profile and customer infrastructure.

10. Restore drill evidence generation is Not currently implemented. Evidence: validation commands exist, but no restore-drill artifact workflow is present.

Low

11. Prometheus TSDB restore is Not currently implemented. Evidence: compose config bounds retention, but no backup/restore procedure is present.

12. Diagnostic bundle and release evidence restore is Not currently implemented. Evidence: retention policy intentionally avoids pruning, but does not define backup or restore automation.

Release-Relevant Conclusion

The repository currently supports operational restart, Windows collector self-healing, local maintenance, retention, readiness evidence, and limited backup-before-mutation behavior.

It does not yet support deterministic full production recovery for all persisted state. The most important gap is not service restart; it is data restore: ActivityWatch DB, ClickHouse volumes, Grafana state, DLP/Hayabusa evidence, and Windows state are not covered by a tested automated restore process in the repository.