AWatch-rus/DISASTER_RECOVERY_PROOF.md

# Disaster Recovery Proof

Date: 2026-07-01

Scope: evidence-backed disaster recovery validation for the current
AWatch-rus / DetMir production deployment, based only on repository contents.
This document does not introduce new recovery procedures and does not change
production behavior.

## Result

Current proof level: partially proven.

Recovery confidence: 48 / 100.

The repository contains deployable components, restart/health checks, Windows
collector recovery automation, ActivityWatch DB merge support, retention guard
rails, and validation scripts. The repository does not yet contain evidence of
a completed separate-host restore for the primary repository/Gitea backup, and
does not implement fully automated restore for all runtime data stores.

Production 1.0 release implication: DR is not blocked by missing documentation,
but remains blocked by missing restore evidence for the repository/Gitea backup
and by missing automated restore coverage for several runtime stores.

## Status Definitions

- Implemented: repository contains executable automation or a concrete
  validation script for the step.
- Partially implemented: repository contains part of the mechanism, but not an
  end-to-end proven path.
- Manual: repository documents the step, but operator execution and evidence
  capture are required.
- Missing: repository evidence shows the step is not implemented.
- Unsupported: repository code explicitly rejects or excludes the operation.

## Evidence Inventory

| Area | Status | Evidence |
| --- | --- | --- |
| General backup/recovery model | Manual | `docs/BACKUP_AND_RECOVERY_RU.md:59-71` defines the generic stop, preserve state, restore, start, health, smoke, and record-result sequence. |
| Production health endpoints | Implemented | `docs/OPERATIONS_RUNBOOK_RU.md:5-13` defines `/healthz`, `/readyz`, and `/metrics` checks; `docs/OPERATIONS_RUNBOOK_RU.md:18-47` defines expected semantics. |
| Post-recovery smoke | Implemented | `docs/OPERATIONS_RUNBOOK_RU.md:49-64` lists smoke scripts and requires them after recovery. |
| Operational maturity validation | Implemented | `docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md:49-59` requires `scripts/public_secret_pattern_check.py`, `scripts/operational-maturity-check.mjs`, and `quality-gate`. |
| Browser/operator validation | Manual | `docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md:64-93` lists required operator pages and secret-handling rules. |
| DetMir production smoke | Implemented | `docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md:95-119` defines `check-aw-full.sh`, `check-aw-data.sh`, and stable `SHARKON2025` host id handling. |
| ActivityWatch service startup | Implemented | `aw-server/activitywatch-server.service:6-14` defines env file, DB path, web path, restart policy, and restart delay. |
| ActivityWatch DB maintenance | Implemented | `aw-server/aw-db-maintenance.service:29-32` and `aw-server/aw-db-maintenance.timer:34-42` define guarded maintenance. |
| Readiness evidence bundle | Implemented | `aw-server/detmir-readiness.service:48-59` and `aw-server/detmir-readiness.timer:60-71` generate daily readiness bundles. |
| Server deploy and API startup validation | Implemented | `ansible/deploy_aw_server.yml:2365-2380` restarts `aw-server/activitywatch-server.service` and waits for `/api/0/info`. |
| ActivityWatch DB merge backup | Partially implemented | `ansible/deploy_aw_server.yml:2222-2350` checks/install merge binary, backs up target and legacy DBs, merges, and installs merged DB when enabled. |
| Rust restore planner | Partially implemented | `scripts/prod_backup_restore.sh:14-29` requires the Rust planner; `adk-rust/crates/prod-backup-restore/src/main.rs:100-105` explicitly rejects `--apply`. |
| Windows package rollback backup | Implemented | `windows/ActivityWatch.Windows.Common.psm1:148-247` backs up install root before replacement and cleans old install backups. |
| Windows recovery loop | Implemented | `windows/ActivityWatch.Windows.Common.psm1:1576-1597` writes recovery script; `windows/ActivityWatch.Windows.Common.psm1:2114-2167` runs the recovery loop. |
| Windows recovery scheduled task | Implemented | `windows/ActivityWatch.Windows.Common.psm1:2447-2495` registers `ActivityWatch Recovery`; `windows/ActivityWatch.Windows.Common.psm1:2656-2672` starts launch tasks and recovery task. |
| Windows deployment validation | Implemented | `ansible/deploy_aw_windows.yml:501-545` checks ActivityWatch API buckets; `ansible/deploy_aw_windows.yml:547-565` runs and fetches endpoint validation. |
| Windows post-deploy validation | Implemented | `ansible/post_validate_aw_windows.yml:55-120` starts recovery/launch tasks and waits for fresh worktime events. |
| Gitea backup | Manual / partially implemented | `docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:7-23` documents path, script, timer, format, checksum, retention, and restore-tested status. |
| Gitea restore proof | Missing | `docs/registry/registry-evidence-manifest.json:26-40` records `restore_tested=false` and `production_ready=false`. |
| ClickHouse 1C runtime store | Partially implemented | `clickhouse-1c/docker-compose.yml:1-19` defines the service and persistent Docker volume; no restore automation is present in the repository. |
| ClickHouse Workforce runtime store | Partially implemented | `clickhouse-workforce/docker-compose.yml:1-17` defines the service and persistent Docker volume; no restore automation is present in the repository. |
| Grafana/Prometheus runtime store | Partially implemented | `grafana-1c/docker-compose.yml:36-79` defines Prometheus/Grafana volumes and Prometheus retention; no Grafana data restore automation is present in the repository. |
| Retention boundaries | Implemented for documented cleanup scopes | `docs/RETENTION_POLICY_RU.md:65-102` documents retention and recovery impact for persistent stores. |

## End-to-End Recovery Chain

| Step | Status | Current evidence | Proof conclusion |
| --- | --- | --- | --- |
| Repository | Partially implemented | Primary self-hosted Gitea repository is recorded in `docs/registry/registry-evidence-manifest.json:6-10`; backup configuration is recorded in `docs/registry/registry-evidence-manifest.json:26-40`. | Source repository is identified and backup metadata exists, but restore proof is missing. |
| Configuration | Manual | Git stores sanitized templates and explicitly excludes secrets/live DBs in `docs/BACKUP_AND_RECOVERY_RU.md:29-46`. | Configuration can be reconstructed from repository templates plus customer secret store, but secrets are intentionally external. |
| Deployment | Implemented | Server deployment restarts ActivityWatch and waits for API readiness in `ansible/deploy_aw_server.yml:2365-2380`; Windows deployment runs package install, recovery tasks, and smoke checks in `ansible/deploy_aw_windows.yml:279-565`. | Server and Windows deployment are executable from repository playbooks. |
| Restore | Partially implemented | AW DB merge path exists in `ansible/deploy_aw_server.yml:2222-2350`; Rust planner emits a plan but rejects `--apply` in `adk-rust/crates/prod-backup-restore/src/main.rs:100-105`; Gitea restore is manual and untested per `docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:42-100`. | Restore is only partially proven. Several runtime stores require manual or future automation. |
| Startup | Implemented | `aw-server/activitywatch-server.service` has restart policy in `aw-server/activitywatch-server.service:6-14`; Windows recovery task and loop are implemented in `windows/ActivityWatch.Windows.Common.psm1:2114-2167` and `:2447-2495`. | Repository supports service startup and collector recovery startup. |
| Health | Implemented | `/healthz`, `/readyz`, `/metrics` are documented in `docs/OPERATIONS_RUNBOOK_RU.md:5-47`; server deployment waits for `/api/0/info` in `ansible/deploy_aw_server.yml:2372-2380`. | Health checks are present and part of deploy/recovery validation. |
| Operational validation | Implemented | Operational gates are documented in `docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md:49-59`; production smoke is documented in `docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md:95-119`. | Repository has repeatable operational validation commands. |
| Pilot validation | Implemented | `docs/OPERATIONS_RUNBOOK_RU.md:49-64` lists pilot/demo/deployment smoke scripts after recovery; `scripts/pilot-validation-smoke.mjs` and `scripts/deployment-readiness-smoke.mjs` exist. | Pilot/deployment smoke validation is implemented as repository scripts. |
| Ready | Manual / conditional | General recovery procedure requires result recording in `docs/BACKUP_AND_RECOVERY_RU.md:59-71`; Gitea manifest still says `production_ready=false` in `docs/registry/registry-evidence-manifest.json:37-40`. | Ready can only be claimed after external restore evidence is captured and manifest gaps are closed. |

## Current Recovery Capability

### Server

Implemented:

- ActivityWatch server restart and API readiness wait through Ansible.
- Systemd restart-on-failure for `aw-server/activitywatch-server.service`.
- Guarded SQLite maintenance and daily readiness bundle timers.
- Limited AW DB merge/recovery-like flow with pre-merge backups.

Partially implemented:

- Generic production restore planning through `prod-backup-restore`, because the
  Rust binary builds a plan but rejects apply mode.
- ActivityWatch DB restore, because the repository implements legacy DB merge,
  but not a generic restore-selected-backup command.

Missing:

- Evidence of a completed end-to-end AW DB restore drill.
- Automated rollback from a failed DB merge to the pre-merge backup.

### Windows/RDP

Implemented:

- Package replacement with install-root backup.
- Recovery script generation.
- Scheduled `ActivityWatch Recovery` task.
- Long-running recovery loop that restarts collectors/tasks for live sessions.
- Post-deploy validation that waits for fresh ActivityWatch worktime events.

Partially implemented:

- Full Windows state restore, because install backups exist but
  `C:\ProgramData\AWatch-rus` restore as a whole is not implemented.

### Repository/Gitea

Manual / partially implemented:

- Backup target, script name, systemd unit/timer names, format, checksum, and
  retention are documented.
- Restore procedure is documented as an outline for a separate server.

Missing:

- Actual separate-host restore evidence.
- Manifest update proving `restore_tested=true`.
- Offsite copy evidence.

### ClickHouse, Grafana, Prometheus

Partially implemented:

- Docker Compose files define persistent volumes and restart policies.
- Prometheus retention is configurable through compose.
- Retention policy documents recovery impact for ClickHouse, Grafana, and
  Prometheus stores.

Missing:

- Automated backup and restore for ClickHouse 1C volume.
- Automated backup and restore for ClickHouse Workforce volume.
- Automated Grafana data restore.
- End-to-end restore evidence for these stores.

## Current Limitations

1. Gitea backup cannot be called production-ready until a separate-host restore
   is performed and recorded. Evidence: `docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:3-5`,
   `docs/registry/registry-evidence-manifest.json:37-40`.
2. `prod-backup-restore --apply` is unsupported by code. Evidence:
   `adk-rust/crates/prod-backup-restore/src/main.rs:100-105`.
3. ActivityWatch DB recovery is limited to legacy-root merge and does not prove
   generic restore from a selected backup file. Evidence:
   `ansible/deploy_aw_server.yml:2222-2350`.
4. ClickHouse/Grafana/Prometheus data volumes are declared, but repository
   restore automation is not present. Evidence:
   `clickhouse-1c/docker-compose.yml:13-19`,
   `clickhouse-workforce/docker-compose.yml:11-17`,
   `grafana-1c/docker-compose.yml:36-79`.
5. Customer secrets are intentionally outside Git. Recovery therefore requires
   access to the customer secret store and cannot be proven from repository
   contents alone. Evidence: `docs/BACKUP_AND_RECOVERY_RU.md:29-46`.
6. Evidence/customer data must not be placed in the public repository. Evidence:
   `docs/BACKUP_AND_RECOVERY_RU.md:91-98`.

## Estimated Recovery Sequence

This sequence is the current evidence-backed chain. Steps marked manual require
operator execution and external evidence capture.

1. Repository availability: clone the primary Gitea repository or validated
   mirror. Status: partially implemented. Evidence:
   `docs/registry/registry-evidence-manifest.json:6-10`.
2. Repository restore, if primary Gitea is lost: follow the Gitea restore
   outline on a separate host, verify checksum, run Gitea checks, and record
   evidence. Status: manual / missing proof. Evidence:
   `docs/registry/GITEA_BACKUP_AND_RESTORE_RUNBOOK_RU.md:42-100`.
3. Configuration recovery: restore sanitized templates from Git and secrets
   from the approved customer secret store. Status: manual. Evidence:
   `docs/BACKUP_AND_RECOVERY_RU.md:29-46`.
4. Server deployment: run server deployment playbook and wait for
   `/api/0/info`. Status: implemented. Evidence:
   `ansible/deploy_aw_server.yml:2365-2380`.
5. AW DB merge path, only when legacy-root merge is explicitly enabled: back up
   target and legacy DBs, merge, install merged DB. Status: partially
   implemented. Evidence: `ansible/deploy_aw_server.yml:2222-2350`.
6. Windows/RDP deployment: run Windows deployment playbook, deploy package,
   configure recovery tasks, and validate bucket events. Status: implemented.
   Evidence: `ansible/deploy_aw_windows.yml:279-565`.
7. Windows post-restore validation: start recovery/launch tasks and wait for
   worktime events. Status: implemented. Evidence:
   `ansible/post_validate_aw_windows.yml:55-120`.
8. Service startup validation: check systemd status, `/healthz`, `/readyz`,
   `/metrics`, and `/api/0/info`. Status: implemented. Evidence:
   `docs/OPERATIONS_RUNBOOK_RU.md:5-47`,
   `ansible/deploy_aw_server.yml:2372-2380`.
9. Production smoke: run `check-aw-full.sh`, `check-aw-data.sh`, and contour
   smoke with stable `SHARKON2025` logical host id. Status: implemented.
   Evidence: `docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md:95-119`.
10. Operational maturity validation: run secret scan, operational maturity
    check, and quality gate. Status: implemented. Evidence:
    `docs/OPERATIONS_VALIDATION_RUNBOOK_RU.md:49-59`.
11. Pilot/deployment readiness validation: run pilot and deployment smoke
    scripts after recovery. Status: implemented. Evidence:
    `docs/OPERATIONS_RUNBOOK_RU.md:49-64`.
12. Ready decision: record recovery result and do not claim production-ready DR
    until missing restore evidence is closed. Status: manual / conditional.
    Evidence: `docs/BACKUP_AND_RECOVERY_RU.md:59-71`,
    `docs/registry/registry-evidence-manifest.json:37-40`.

## Evidence Gaps

| Gap | Status | Required evidence before closing |
| --- | --- | --- |
| Gitea separate-host restore | Missing | Backup filename, SHA256 verification output, Gitea version, restore duration, post-restore checks, clone/access proof, and manifest update. |
| Generic AW DB restore | Missing | Tested command or runbook restoring a selected backup into active DB with checksum and rollback evidence. |
| ClickHouse 1C restore | Missing | Tested volume/table backup and restore evidence for `clickhouse_1c_data`. |
| ClickHouse Workforce restore | Missing | Tested volume/table backup and restore evidence for `clickhouse_workforce_data`. |
| Grafana data restore | Missing | Tested restore evidence for `grafana-data` or explicit proof that provisioned dashboards plus documented credentials are sufficient. |
| Off-host backup copy | Missing | Repository evidence of destination, retention, checksum policy, access policy, and restore test. |

## Recommended Future Automation

These are future tasks, not implemented by this document:

1. Add a non-destructive restore drill checklist artifact that records exact
   backup filename, checksum, host, duration, and post-restore checks.
2. Add a separate-host Gitea restore evidence template and only then update
   `restore_tested` in `docs/registry/registry-evidence-manifest.json`.
3. Add tested backup/restore automation for ClickHouse 1C and Workforce volumes
   or explicitly document that those stores are rebuilt from source exports.
4. Add a generic ActivityWatch DB restore runbook with rollback command and
   checksum verification.
5. Add a single post-restore validation command that runs health, operational
   maturity, pilot validation, and deployment readiness in the documented order.

## Release Decision

DR proof is sufficient to show that the repository contains significant
recoverability mechanisms and validation gates.

DR proof is not sufficient to claim full Production 1.0 disaster recovery until
the missing separate-host restore evidence and runtime store restore gaps are
closed.