Add DetMir primary recovery guard
This commit is contained in:
igor04091968
@@ -250,6 +250,10 @@ server {
|
|||||||
}
|
}
|
||||||
|
|
||||||
{% for route in proxmox_web_gateway_routes %}
|
{% for route in proxmox_web_gateway_routes %}
|
||||||
|
location = /go/{{ route.slug }} {
|
||||||
|
return 302 {{ route.target_url }};
|
||||||
|
}
|
||||||
|
|
||||||
{% if route.proxy_target_url is defined %}
|
{% if route.proxy_target_url is defined %}
|
||||||
location = {{ route.proxy_path }} {
|
location = {{ route.proxy_path }} {
|
||||||
proxy_set_header Authorization "";
|
proxy_set_header Authorization "";
|
||||||
@@ -274,7 +278,7 @@ server {
|
|||||||
location ^~ /portal/api/readiness {
|
location ^~ /portal/api/readiness {
|
||||||
proxy_set_header Authorization "";
|
proxy_set_header Authorization "";
|
||||||
proxy_set_header X-Remote-User $remote_user;
|
proxy_set_header X-Remote-User $remote_user;
|
||||||
proxy_pass http://192.0.2.13:8721/api/readiness;
|
proxy_pass http://127.0.0.1:8720/readyz;
|
||||||
proxy_redirect off;
|
proxy_redirect off;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Ежедневное обслуживание AWatch-rus
|
# Ежедневное обслуживание AWatch-rus
|
||||||
|
|
||||||
Дата актуализации: 2026-07-02
|
Дата актуализации: 2026-07-03
|
||||||
|
|
||||||
Документ описывает ежедневный операторский цикл обслуживания AWatch-rus /
|
Документ описывает ежедневный операторский цикл обслуживания AWatch-rus /
|
||||||
DetMir и безопасное использование Pollinations AI как вспомогательного
|
DetMir и безопасное использование Pollinations AI как вспомогательного
|
||||||
@@ -87,6 +87,22 @@ node scripts/deployment-readiness-smoke.mjs
|
|||||||
Критерий: нет новых hard-fail проверок. Warning допустим только при известном
|
Критерий: нет новых hard-fail проверок. Warning допустим только при известном
|
||||||
и задокументированном operational constraint.
|
и задокументированном operational constraint.
|
||||||
|
|
||||||
|
### 3.1. Primary recovery guard
|
||||||
|
|
||||||
|
Проверить, что автоматический recovery первичного AW API включен и не скрывает
|
||||||
|
новые incident:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl is-active detmir-aw-primary-recovery.timer
|
||||||
|
systemctl status detmir-aw-primary-recovery.timer --no-pager
|
||||||
|
sudo /usr/local/bin/detmir-aw-primary-recovery --check-only
|
||||||
|
sudo jq . /var/lib/detmir-aw-primary-recovery/latest.json 2>/dev/null || true
|
||||||
|
```
|
||||||
|
|
||||||
|
Критерий: timer активен, `--check-only` возвращает `status=ok`, новые incident
|
||||||
|
не имеют `outcome=failed`. Если latest incident отсутствует, это нормально для
|
||||||
|
контура, где recovery еще ни разу не требовался.
|
||||||
|
|
||||||
### 4. Очереди, backlog и storage growth
|
### 4. Очереди, backlog и storage growth
|
||||||
|
|
||||||
Проверить отсутствие неконтролируемого роста:
|
Проверить отсутствие неконтролируемого роста:
|
||||||
@@ -305,6 +321,7 @@ unset ALL_PROXY all_proxy
|
|||||||
|
|
||||||
- [Operations Runbook](OPERATIONS_RUNBOOK_RU.md)
|
- [Operations Runbook](OPERATIONS_RUNBOOK_RU.md)
|
||||||
- [Эксплуатационная проверка контура](OPERATIONS_VALIDATION_RUNBOOK_RU.md)
|
- [Эксплуатационная проверка контура](OPERATIONS_VALIDATION_RUNBOOK_RU.md)
|
||||||
|
- [DetMir service reliability runbook](DETMIR_SERVICE_RELIABILITY_RUNBOOK_RU.md)
|
||||||
- [Retention and Cleanup Policy](RETENTION_POLICY_RU.md)
|
- [Retention and Cleanup Policy](RETENTION_POLICY_RU.md)
|
||||||
- [Production readiness](PRODUCTION_READINESS_RU.md)
|
- [Production readiness](PRODUCTION_READINESS_RU.md)
|
||||||
- [Autonomous validation](AUTONOMOUS_VALIDATION_RU.md)
|
- [Autonomous validation](AUTONOMOUS_VALIDATION_RU.md)
|
||||||
|
|||||||
@@ -0,0 +1,233 @@
|
|||||||
|
# DetMir Service Reliability Runbook
|
||||||
|
|
||||||
|
Дата актуализации: 2026-07-03
|
||||||
|
|
||||||
|
Документ описывает production-safe контур снижения повторных выпадений сервисов
|
||||||
|
AWatch-rus/DetMir. Главная цель — отделять первичный отказ ActivityWatch API от
|
||||||
|
вторичных падений ingest/report/health jobs и выполнять ограниченное
|
||||||
|
автовосстановление только для подтвержденного безопасного сценария.
|
||||||
|
|
||||||
|
## План реализации
|
||||||
|
|
||||||
|
1. Считать `activitywatch-server` первичным сервисом для AW API.
|
||||||
|
2. Считать `aw-workforce-ingest`, `aw-worktime-autoheal`, `aw-worktime-prewarm`,
|
||||||
|
`aw-worktime-ui-bridge`, `aw-worktime-influx-exporter`, `aw-rus-healthd` и
|
||||||
|
`aw-slo-monitor` вторичными зависимыми jobs.
|
||||||
|
3. Проверять AW API через `/api/0/settings/` и `/api/0/buckets/`.
|
||||||
|
4. Автоматически восстанавливать только подтвержденный класс отказа:
|
||||||
|
`poisoned datastore lock` или неактивный `activitywatch-server`.
|
||||||
|
5. Перед restart первичного сервиса временно остановить вторичные jobs, чтобы
|
||||||
|
они не усиливали нагрузку на AW SQLite.
|
||||||
|
6. Перезапустить только `activitywatch-server` внутри CT `203`.
|
||||||
|
7. Дождаться восстановления AW API.
|
||||||
|
8. Вернуть вторичные timers и записать incident JSON.
|
||||||
|
|
||||||
|
## Что реализовано
|
||||||
|
|
||||||
|
Реализован script:
|
||||||
|
|
||||||
|
```text
|
||||||
|
scripts/detmir-aw-primary-recovery.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Systemd units для Proxmox host:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ops/systemd/detmir-aw-primary-recovery.service
|
||||||
|
ops/systemd/detmir-aw-primary-recovery.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
Контур предназначен для запуска на Proxmox host `10.10.10.2`, потому что там
|
||||||
|
доступны одновременно:
|
||||||
|
|
||||||
|
- управление PVE jobs через локальный `systemctl`;
|
||||||
|
- управление AW CT через `pct exec 203`;
|
||||||
|
- проверка внешнего AW API `http://10.10.10.13:5600`.
|
||||||
|
|
||||||
|
## Safety Rules
|
||||||
|
|
||||||
|
Recovery script:
|
||||||
|
|
||||||
|
- не удаляет SQLite database, lock files или journal files;
|
||||||
|
- не включает DLP, Loki или always-on Velociraptor;
|
||||||
|
- не перезапускает Windows host и Windows collectors;
|
||||||
|
- не меняет конфигурацию AW server;
|
||||||
|
- не выполняет recovery при произвольном HTTP timeout без подтвержденной
|
||||||
|
причины;
|
||||||
|
- имеет cooldown между restart попытками;
|
||||||
|
- пишет incident evidence в JSON.
|
||||||
|
|
||||||
|
## Установка
|
||||||
|
|
||||||
|
На Proxmox host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo install -m 0755 scripts/detmir-aw-primary-recovery.sh \
|
||||||
|
/usr/local/bin/detmir-aw-primary-recovery
|
||||||
|
|
||||||
|
sudo install -m 0644 ops/systemd/detmir-aw-primary-recovery.service \
|
||||||
|
/etc/systemd/system/detmir-aw-primary-recovery.service
|
||||||
|
|
||||||
|
sudo install -m 0644 ops/systemd/detmir-aw-primary-recovery.timer \
|
||||||
|
/etc/systemd/system/detmir-aw-primary-recovery.timer
|
||||||
|
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable --now detmir-aw-primary-recovery.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
Опциональная конфигурация:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo install -d -m 0755 /etc/detmir
|
||||||
|
sudoedit /etc/detmir/aw-primary-recovery.env
|
||||||
|
```
|
||||||
|
|
||||||
|
Пример:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
DETMIR_AW_RECOVERY_URL=http://10.10.10.13:5600
|
||||||
|
DETMIR_AW_RECOVERY_CT_ID=203
|
||||||
|
DETMIR_AW_RECOVERY_CONFIRM_ATTEMPTS=2
|
||||||
|
DETMIR_AW_RECOVERY_COOLDOWN_SECONDS=900
|
||||||
|
DETMIR_AW_RECOVERY_HTTP_TIMEOUT_SECONDS=8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Проверка без изменений
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/usr/local/bin/detmir-aw-primary-recovery --self-test
|
||||||
|
/usr/local/bin/detmir-aw-primary-recovery --check-only
|
||||||
|
systemctl start detmir-aw-primary-recovery.service
|
||||||
|
journalctl -u detmir-aw-primary-recovery.service -n 80 --no-pager
|
||||||
|
```
|
||||||
|
|
||||||
|
Ожидаемо для здорового контура:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=probe attempt=1 status=ok
|
||||||
|
```
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
Incident files:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/var/lib/detmir-aw-primary-recovery/incidents/*.json
|
||||||
|
/var/lib/detmir-aw-primary-recovery/latest.json
|
||||||
|
/var/lib/detmir-aw-primary-recovery/status.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Проверка последнего incident:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo jq . /var/lib/detmir-aw-primary-recovery/latest.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Ключевые поля:
|
||||||
|
|
||||||
|
- `outcome`: `recovered`, `failed`, `skipped`, `observed_no_action`, `dry_run`;
|
||||||
|
- `reason`: `poisoned_lock`, `service_inactive`, `cooldown_active_after_*`;
|
||||||
|
- `recovered`: boolean;
|
||||||
|
- `pve_pause_units`;
|
||||||
|
- `ct_pause_units`;
|
||||||
|
- `details`.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Отключить automation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl disable --now detmir-aw-primary-recovery.timer
|
||||||
|
sudo systemctl reset-failed detmir-aw-primary-recovery.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Удаление установленных файлов не требуется для rollback. Если нужно убрать
|
||||||
|
полностью:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo rm -f /etc/systemd/system/detmir-aw-primary-recovery.service
|
||||||
|
sudo rm -f /etc/systemd/system/detmir-aw-primary-recovery.timer
|
||||||
|
sudo rm -f /usr/local/bin/detmir-aw-primary-recovery
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
```
|
||||||
|
|
||||||
|
## Manual Recovery Sequence
|
||||||
|
|
||||||
|
Если automation disabled или recovery не помог:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl stop aw-workforce-ingest.timer aw-workforce-ingest.service
|
||||||
|
sudo pct exec 203 -- systemctl stop \
|
||||||
|
aw-worktime-autoheal.timer aw-worktime-autoheal.service \
|
||||||
|
aw-worktime-prewarm.timer aw-worktime-prewarm.service \
|
||||||
|
aw-worktime-ui-bridge.timer aw-worktime-ui-bridge.service \
|
||||||
|
aw-worktime-influx-exporter.timer aw-worktime-influx-exporter.service \
|
||||||
|
aw-rus-healthd.timer aw-rus-healthd.service \
|
||||||
|
aw-slo-monitor.timer aw-slo-monitor.service
|
||||||
|
|
||||||
|
sudo pct exec 203 -- systemctl restart activitywatch-server
|
||||||
|
|
||||||
|
curl -fsS http://10.10.10.13:5600/api/0/settings/ >/dev/null
|
||||||
|
curl -fsS http://10.10.10.13:5600/api/0/buckets/ >/dev/null
|
||||||
|
|
||||||
|
sudo pct exec 203 -- systemctl start \
|
||||||
|
aw-worktime-autoheal.timer \
|
||||||
|
aw-worktime-prewarm.timer \
|
||||||
|
aw-worktime-ui-bridge.timer \
|
||||||
|
aw-worktime-influx-exporter.timer \
|
||||||
|
aw-rus-healthd.timer \
|
||||||
|
aw-slo-monitor.timer
|
||||||
|
|
||||||
|
sudo systemctl start aw-workforce-ingest.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
## Post-Recovery Validation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /mnt/usb_hdd2/Projects/ActivityWatch-Russian
|
||||||
|
./check-aw-full.sh
|
||||||
|
|
||||||
|
ssh igor@10.10.10.2 'systemctl --failed --no-legend || true'
|
||||||
|
ssh igor@10.10.10.2 'sudo pct exec 203 -- systemctl --failed --no-legend || true'
|
||||||
|
|
||||||
|
curl -fsS http://10.10.10.13:5610/health | jq .
|
||||||
|
curl -fsS http://10.10.10.2:8720/readyz | jq .
|
||||||
|
```
|
||||||
|
|
||||||
|
Критерий:
|
||||||
|
|
||||||
|
- `check-aw-full.sh`: `DEAD=0`, `STALE=0`;
|
||||||
|
- failed units на Proxmox host: `0`;
|
||||||
|
- failed units внутри CT `203`: `0`;
|
||||||
|
- Worktime health: `status=OK`;
|
||||||
|
- Portal readiness: `status=ready`.
|
||||||
|
|
||||||
|
## Gateway Contract
|
||||||
|
|
||||||
|
Операторские shortcuts должны соответствовать runbook:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/go/proxmox-gui
|
||||||
|
/go/file1c-brief
|
||||||
|
/go/file1c-actions
|
||||||
|
```
|
||||||
|
|
||||||
|
Они формируются из `proxmox_web_gateway_routes` в nginx template и должны
|
||||||
|
возвращать `302` на целевой внутренний URL после Basic Auth.
|
||||||
|
|
||||||
|
Readiness compatibility endpoint:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/portal/api/readiness -> http://127.0.0.1:8720/readyz
|
||||||
|
```
|
||||||
|
|
||||||
|
## Known Limitations
|
||||||
|
|
||||||
|
- Recovery intentionally limited to AW API poisoned lock and inactive primary
|
||||||
|
service. Other failures remain operator-visible incidents.
|
||||||
|
- If restart does not restore AW API before timeout, secondary timers remain
|
||||||
|
paused to avoid failure amplification. Operator must inspect
|
||||||
|
`/var/lib/detmir-aw-primary-recovery/latest.json`.
|
||||||
|
- Windows credential or WinRM authentication problems are outside this recovery
|
||||||
|
loop.
|
||||||
|
- This does not prove backup/restore. It is runtime availability recovery only.
|
||||||
@@ -6,6 +6,10 @@
|
|||||||
использование Pollinations AI для анализа sanitized evidence описаны отдельно:
|
использование Pollinations AI для анализа sanitized evidence описаны отдельно:
|
||||||
[ежедневное обслуживание AWatch-rus](DAILY_MAINTENANCE_RU.md).
|
[ежедневное обслуживание AWatch-rus](DAILY_MAINTENANCE_RU.md).
|
||||||
|
|
||||||
|
Автоматическое восстановление первичного AW API от подтвержденного
|
||||||
|
`poisoned datastore lock` описано отдельно:
|
||||||
|
[DetMir service reliability runbook](DETMIR_SERVICE_RELIABILITY_RUNBOOK_RU.md).
|
||||||
|
|
||||||
## Быстрая проверка
|
## Быстрая проверка
|
||||||
|
|
||||||
Проверить доступность:
|
Проверить доступность:
|
||||||
@@ -108,6 +112,20 @@ systemctl --failed --no-pager
|
|||||||
- coverage;
|
- coverage;
|
||||||
- errors in logs.
|
- errors in logs.
|
||||||
|
|
||||||
|
### ActivityWatch API возвращает `503 poisoned lock`
|
||||||
|
|
||||||
|
Проверить, сработал ли primary recovery guard:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl status detmir-aw-primary-recovery.timer --no-pager
|
||||||
|
journalctl -u detmir-aw-primary-recovery.service -n 80 --no-pager
|
||||||
|
sudo jq . /var/lib/detmir-aw-primary-recovery/latest.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Если guard отключен, использовать manual sequence из
|
||||||
|
`DETMIR_SERVICE_RELIABILITY_RUNBOOK_RU.md`. Не удалять SQLite, lock или journal
|
||||||
|
файлы ActivityWatch вручную.
|
||||||
|
|
||||||
### Нет данных
|
### Нет данных
|
||||||
|
|
||||||
Проверить:
|
Проверить:
|
||||||
|
|||||||
@@ -0,0 +1,18 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=DetMir ActivityWatch primary recovery guard
|
||||||
|
Documentation=https://github.com/igor04091968/AWatch-rus/blob/main/docs/DETMIR_SERVICE_RELIABILITY_RUNBOOK_RU.md
|
||||||
|
Wants=network-online.target
|
||||||
|
After=network-online.target
|
||||||
|
ConditionPathExists=/usr/sbin/pct
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
EnvironmentFile=-/etc/detmir/aw-primary-recovery.env
|
||||||
|
ExecStart=/usr/local/bin/detmir-aw-primary-recovery --once
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
Nice=5
|
||||||
|
TimeoutStartSec=5min
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
SyslogIdentifier=detmir-aw-primary-recovery
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Run DetMir ActivityWatch primary recovery guard every 2 minutes
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=3min
|
||||||
|
OnUnitActiveSec=2min
|
||||||
|
AccuracySec=30s
|
||||||
|
RandomizedDelaySec=15s
|
||||||
|
Persistent=false
|
||||||
|
Unit=detmir-aw-primary-recovery.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
Executable
+367
@@ -0,0 +1,367 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Production-safe primary recovery loop for DetMir ActivityWatch.
|
||||||
|
#
|
||||||
|
# Scope:
|
||||||
|
# - Detects ActivityWatch API poisoned datastore lock.
|
||||||
|
# - Pauses dependent secondary jobs before restarting the primary AW service.
|
||||||
|
# - Restarts only activitywatch-server inside the configured Proxmox CT.
|
||||||
|
# - Writes structured incident evidence.
|
||||||
|
#
|
||||||
|
# Non-goals:
|
||||||
|
# - Does not delete SQLite, journal or lock files.
|
||||||
|
# - Does not enable DLP, Loki or Velociraptor.
|
||||||
|
# - Does not restart Windows collectors.
|
||||||
|
|
||||||
|
AW_URL="${DETMIR_AW_RECOVERY_URL:-http://10.10.10.13:5600}"
|
||||||
|
AW_CT_ID="${DETMIR_AW_RECOVERY_CT_ID:-203}"
|
||||||
|
AW_SERVICE="${DETMIR_AW_RECOVERY_SERVICE:-activitywatch-server}"
|
||||||
|
STATE_DIR="${DETMIR_AW_RECOVERY_STATE_DIR:-/var/lib/detmir-aw-primary-recovery}"
|
||||||
|
HTTP_TIMEOUT_SECONDS="${DETMIR_AW_RECOVERY_HTTP_TIMEOUT_SECONDS:-8}"
|
||||||
|
CONFIRM_ATTEMPTS="${DETMIR_AW_RECOVERY_CONFIRM_ATTEMPTS:-2}"
|
||||||
|
CONFIRM_SLEEP_SECONDS="${DETMIR_AW_RECOVERY_CONFIRM_SLEEP_SECONDS:-5}"
|
||||||
|
STARTUP_TIMEOUT_SECONDS="${DETMIR_AW_RECOVERY_STARTUP_TIMEOUT_SECONDS:-120}"
|
||||||
|
STARTUP_SLEEP_SECONDS="${DETMIR_AW_RECOVERY_STARTUP_SLEEP_SECONDS:-5}"
|
||||||
|
COOLDOWN_SECONDS="${DETMIR_AW_RECOVERY_COOLDOWN_SECONDS:-900}"
|
||||||
|
DRY_RUN="${DETMIR_AW_RECOVERY_DRY_RUN:-0}"
|
||||||
|
|
||||||
|
PVE_PAUSE_UNITS="${DETMIR_AW_RECOVERY_PVE_PAUSE_UNITS:-aw-workforce-ingest.timer aw-workforce-ingest.service}"
|
||||||
|
PVE_RESUME_UNITS="${DETMIR_AW_RECOVERY_PVE_RESUME_UNITS:-aw-workforce-ingest.timer}"
|
||||||
|
CT_PAUSE_UNITS="${DETMIR_AW_RECOVERY_CT_PAUSE_UNITS:-aw-worktime-autoheal.timer aw-worktime-autoheal.service aw-worktime-prewarm.timer aw-worktime-prewarm.service aw-worktime-ui-bridge.timer aw-worktime-ui-bridge.service aw-worktime-influx-exporter.timer aw-worktime-influx-exporter.service aw-rus-healthd.timer aw-rus-healthd.service aw-slo-monitor.timer aw-slo-monitor.service}"
|
||||||
|
CT_RESUME_UNITS="${DETMIR_AW_RECOVERY_CT_RESUME_UNITS:-aw-worktime-autoheal.timer aw-worktime-prewarm.timer aw-worktime-ui-bridge.timer aw-worktime-influx-exporter.timer aw-rus-healthd.timer aw-slo-monitor.timer}"
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<'EOF'
|
||||||
|
Usage:
|
||||||
|
detmir-aw-primary-recovery.sh [--once|--check-only|--self-test]
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
DETMIR_AW_RECOVERY_URL default: http://10.10.10.13:5600
|
||||||
|
DETMIR_AW_RECOVERY_CT_ID default: 203
|
||||||
|
DETMIR_AW_RECOVERY_SERVICE default: activitywatch-server
|
||||||
|
DETMIR_AW_RECOVERY_STATE_DIR default: /var/lib/detmir-aw-primary-recovery
|
||||||
|
DETMIR_AW_RECOVERY_CONFIRM_ATTEMPTS default: 2
|
||||||
|
DETMIR_AW_RECOVERY_COOLDOWN_SECONDS default: 900
|
||||||
|
DETMIR_AW_RECOVERY_DRY_RUN default: 0
|
||||||
|
DETMIR_AW_RECOVERY_PVE_PAUSE_UNITS space-separated units on Proxmox host
|
||||||
|
DETMIR_AW_RECOVERY_PVE_RESUME_UNITS space-separated units on Proxmox host
|
||||||
|
DETMIR_AW_RECOVERY_CT_PAUSE_UNITS space-separated units inside AW CT
|
||||||
|
DETMIR_AW_RECOVERY_CT_RESUME_UNITS space-separated units inside AW CT
|
||||||
|
|
||||||
|
The script never removes ActivityWatch SQLite, lock or journal files.
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
log() {
|
||||||
|
printf 'ts=%s component=detmir-aw-primary-recovery %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*"
|
||||||
|
}
|
||||||
|
|
||||||
|
require_command() {
|
||||||
|
local cmd="$1"
|
||||||
|
if ! command -v "$cmd" >/dev/null 2>&1; then
|
||||||
|
log "level=error event=missing_command command=$cmd"
|
||||||
|
exit 127
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
json_string_array() {
|
||||||
|
jq -Rsc 'split(" ") | map(select(length > 0))' <<<"$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
record_incident() {
|
||||||
|
local outcome="$1"
|
||||||
|
local reason="$2"
|
||||||
|
local recovered="$3"
|
||||||
|
local started_at="$4"
|
||||||
|
local finished_at="$5"
|
||||||
|
local details="$6"
|
||||||
|
local incident_id
|
||||||
|
incident_id="$(date -u +%Y%m%dT%H%M%SZ)-$$"
|
||||||
|
local incident_path="${STATE_DIR}/incidents/${incident_id}.json"
|
||||||
|
install -d -m 0750 "${STATE_DIR}/incidents"
|
||||||
|
jq -n \
|
||||||
|
--arg generated_at_utc "$finished_at" \
|
||||||
|
--arg started_at_utc "$started_at" \
|
||||||
|
--arg outcome "$outcome" \
|
||||||
|
--arg reason "$reason" \
|
||||||
|
--arg recovered "$recovered" \
|
||||||
|
--arg aw_url "$AW_URL" \
|
||||||
|
--arg aw_ct_id "$AW_CT_ID" \
|
||||||
|
--arg aw_service "$AW_SERVICE" \
|
||||||
|
--arg details "$details" \
|
||||||
|
--argjson pve_pause_units "$(json_string_array "$PVE_PAUSE_UNITS")" \
|
||||||
|
--argjson ct_pause_units "$(json_string_array "$CT_PAUSE_UNITS")" \
|
||||||
|
'{
|
||||||
|
generated_at_utc: $generated_at_utc,
|
||||||
|
started_at_utc: $started_at_utc,
|
||||||
|
outcome: $outcome,
|
||||||
|
reason: $reason,
|
||||||
|
recovered: ($recovered == "true"),
|
||||||
|
aw_url: $aw_url,
|
||||||
|
aw_ct_id: ($aw_ct_id | tonumber? // $aw_ct_id),
|
||||||
|
aw_service: $aw_service,
|
||||||
|
details: $details,
|
||||||
|
pve_pause_units: $pve_pause_units,
|
||||||
|
ct_pause_units: $ct_pause_units
|
||||||
|
}' >"$incident_path"
|
||||||
|
ln -sfn "$incident_path" "${STATE_DIR}/latest.json"
|
||||||
|
log "level=info event=incident_written path=$incident_path outcome=$outcome reason=$reason recovered=$recovered"
|
||||||
|
}
|
||||||
|
|
||||||
|
classify_http_response() {
|
||||||
|
local code="$1"
|
||||||
|
local body_file="$2"
|
||||||
|
if [[ "$code" =~ ^2[0-9][0-9]$ ]]; then
|
||||||
|
printf 'ok'
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if grep -Eiq 'poisoned lock|Taking datastore lock failed|datastore lock failed' "$body_file"; then
|
||||||
|
printf 'poisoned_lock'
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
printf 'http_%s' "$code"
|
||||||
|
}
|
||||||
|
|
||||||
|
probe_path() {
|
||||||
|
local path="$1"
|
||||||
|
local body_file
|
||||||
|
body_file="$(mktemp)"
|
||||||
|
local code rc
|
||||||
|
code="$(curl -sS --max-time "$HTTP_TIMEOUT_SECONDS" -o "$body_file" -w '%{http_code}' "${AW_URL}${path}" 2>/dev/null)" || rc=$?
|
||||||
|
rc="${rc:-0}"
|
||||||
|
if [[ "$rc" -ne 0 ]]; then
|
||||||
|
rm -f "$body_file"
|
||||||
|
printf 'timeout'
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local status
|
||||||
|
status="$(classify_http_response "$code" "$body_file")"
|
||||||
|
rm -f "$body_file"
|
||||||
|
printf '%s' "$status"
|
||||||
|
}
|
||||||
|
|
||||||
|
ct_systemctl() {
|
||||||
|
pct exec "$AW_CT_ID" -- systemctl "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
activitywatch_service_status() {
|
||||||
|
if ! pct status "$AW_CT_ID" >/dev/null 2>&1; then
|
||||||
|
printf 'ct_unavailable'
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if ct_systemctl is-active --quiet "$AW_SERVICE"; then
|
||||||
|
printf 'active'
|
||||||
|
else
|
||||||
|
printf 'inactive'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
probe_aw() {
|
||||||
|
local service_state settings_status buckets_status
|
||||||
|
service_state="$(activitywatch_service_status)"
|
||||||
|
if [[ "$service_state" != "active" ]]; then
|
||||||
|
printf 'service_%s' "$service_state"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
settings_status="$(probe_path '/api/0/settings/')"
|
||||||
|
buckets_status="$(probe_path '/api/0/buckets/')"
|
||||||
|
if [[ "$settings_status" == "ok" && "$buckets_status" == "ok" ]]; then
|
||||||
|
printf 'ok'
|
||||||
|
elif [[ "$settings_status" == "poisoned_lock" || "$buckets_status" == "poisoned_lock" ]]; then
|
||||||
|
printf 'poisoned_lock'
|
||||||
|
else
|
||||||
|
printf '%s,%s' "$settings_status" "$buckets_status"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_units_on_host() {
|
||||||
|
local units="$1"
|
||||||
|
local unit
|
||||||
|
for unit in $units; do
|
||||||
|
log "level=info event=stop_unit scope=pve unit=$unit"
|
||||||
|
systemctl stop "$unit" >/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
start_units_on_host() {
|
||||||
|
local units="$1"
|
||||||
|
local unit
|
||||||
|
for unit in $units; do
|
||||||
|
log "level=info event=start_unit scope=pve unit=$unit"
|
||||||
|
systemctl start "$unit" >/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_units_in_ct() {
|
||||||
|
local units="$1"
|
||||||
|
local unit
|
||||||
|
for unit in $units; do
|
||||||
|
log "level=info event=stop_unit scope=ct ct=$AW_CT_ID unit=$unit"
|
||||||
|
ct_systemctl stop "$unit" >/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
start_units_in_ct() {
|
||||||
|
local units="$1"
|
||||||
|
local unit
|
||||||
|
for unit in $units; do
|
||||||
|
log "level=info event=start_unit scope=ct ct=$AW_CT_ID unit=$unit"
|
||||||
|
ct_systemctl start "$unit" >/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
cooldown_active() {
|
||||||
|
local last_file="${STATE_DIR}/last_restart_epoch"
|
||||||
|
[[ -f "$last_file" ]] || return 1
|
||||||
|
local now last
|
||||||
|
now="$(date -u +%s)"
|
||||||
|
last="$(cat "$last_file" 2>/dev/null || printf '0')"
|
||||||
|
[[ "$last" =~ ^[0-9]+$ ]] || return 1
|
||||||
|
(( now - last < COOLDOWN_SECONDS ))
|
||||||
|
}
|
||||||
|
|
||||||
|
wait_for_aw_ok() {
|
||||||
|
local deadline now status
|
||||||
|
deadline=$(( $(date -u +%s) + STARTUP_TIMEOUT_SECONDS ))
|
||||||
|
while true; do
|
||||||
|
status="$(probe_aw)"
|
||||||
|
if [[ "$status" == "ok" ]]; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
now="$(date -u +%s)"
|
||||||
|
if (( now >= deadline )); then
|
||||||
|
log "level=error event=wait_for_aw_timeout last_status=$status"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
log "level=info event=wait_for_aw status=$status"
|
||||||
|
sleep "$STARTUP_SLEEP_SECONDS"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
run_recovery() {
|
||||||
|
local reason="$1"
|
||||||
|
local started_at="$2"
|
||||||
|
if cooldown_active; then
|
||||||
|
local finished_at
|
||||||
|
finished_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
record_incident "skipped" "cooldown_active_after_${reason}" "false" "$started_at" "$finished_at" "restart suppressed by cooldown"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == "1" ]]; then
|
||||||
|
local finished_at
|
||||||
|
finished_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
record_incident "dry_run" "$reason" "false" "$started_at" "$finished_at" "dry run requested"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "level=warn event=recovery_start reason=$reason ct=$AW_CT_ID service=$AW_SERVICE"
|
||||||
|
stop_units_on_host "$PVE_PAUSE_UNITS"
|
||||||
|
stop_units_in_ct "$CT_PAUSE_UNITS"
|
||||||
|
|
||||||
|
log "level=warn event=restart_primary ct=$AW_CT_ID service=$AW_SERVICE"
|
||||||
|
ct_systemctl restart "$AW_SERVICE"
|
||||||
|
printf '%s\n' "$(date -u +%s)" >"${STATE_DIR}/last_restart_epoch"
|
||||||
|
|
||||||
|
local finished_at
|
||||||
|
if wait_for_aw_ok; then
|
||||||
|
start_units_in_ct "$CT_RESUME_UNITS"
|
||||||
|
start_units_on_host "$PVE_RESUME_UNITS"
|
||||||
|
finished_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
record_incident "recovered" "$reason" "true" "$started_at" "$finished_at" "primary restarted and AW API returned ok"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
finished_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
record_incident "failed" "$reason" "false" "$started_at" "$finished_at" "primary restart did not restore AW API; secondary timers left paused"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
check_once() {
|
||||||
|
install -d -m 0750 "$STATE_DIR"
|
||||||
|
local started_at status attempt reason
|
||||||
|
started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
reason=""
|
||||||
|
for attempt in $(seq 1 "$CONFIRM_ATTEMPTS"); do
|
||||||
|
status="$(probe_aw)"
|
||||||
|
log "level=info event=probe attempt=$attempt status=$status"
|
||||||
|
case "$status" in
|
||||||
|
ok)
|
||||||
|
jq -n \
|
||||||
|
--arg generated_at_utc "$started_at" \
|
||||||
|
--arg status "ok" \
|
||||||
|
--arg aw_url "$AW_URL" \
|
||||||
|
'{generated_at_utc: $generated_at_utc, status: $status, aw_url: $aw_url}' \
|
||||||
|
>"${STATE_DIR}/status.json"
|
||||||
|
return 0
|
||||||
|
;;
|
||||||
|
poisoned_lock|service_inactive|service_ct_unavailable)
|
||||||
|
reason="$status"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
local finished_at
|
||||||
|
finished_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
record_incident "observed_no_action" "$status" "false" "$started_at" "$finished_at" "unhealthy state is not in automatic recovery allowlist"
|
||||||
|
return 0
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
if [[ "$attempt" -lt "$CONFIRM_ATTEMPTS" ]]; then
|
||||||
|
sleep "$CONFIRM_SLEEP_SECONDS"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
run_recovery "$reason" "$started_at"
|
||||||
|
}
|
||||||
|
|
||||||
|
self_test() {
|
||||||
|
local tmp
|
||||||
|
tmp="$(mktemp)"
|
||||||
|
printf '{"message":"Taking datastore lock failed, returning 504: poisoned lock: another task failed inside"}' >"$tmp"
|
||||||
|
[[ "$(classify_http_response 503 "$tmp")" == "poisoned_lock" ]]
|
||||||
|
printf '{"ok":true}' >"$tmp"
|
||||||
|
[[ "$(classify_http_response 200 "$tmp")" == "ok" ]]
|
||||||
|
printf '{"message":"other"}' >"$tmp"
|
||||||
|
[[ "$(classify_http_response 503 "$tmp")" == "http_503" ]]
|
||||||
|
rm -f "$tmp"
|
||||||
|
log "level=info event=self_test status=ok"
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
local mode="${1:---once}"
|
||||||
|
case "$mode" in
|
||||||
|
--help|-h)
|
||||||
|
usage
|
||||||
|
;;
|
||||||
|
--self-test)
|
||||||
|
require_command jq
|
||||||
|
self_test
|
||||||
|
;;
|
||||||
|
--check-only)
|
||||||
|
require_command curl
|
||||||
|
require_command jq
|
||||||
|
require_command pct
|
||||||
|
status="$(probe_aw)"
|
||||||
|
log "level=info event=check_only status=$status"
|
||||||
|
[[ "$status" == "ok" ]]
|
||||||
|
;;
|
||||||
|
--once)
|
||||||
|
require_command curl
|
||||||
|
require_command jq
|
||||||
|
require_command pct
|
||||||
|
require_command flock
|
||||||
|
install -d -m 0750 "$STATE_DIR"
|
||||||
|
exec 9>"${STATE_DIR}/lock"
|
||||||
|
if ! flock -n 9; then
|
||||||
|
log "level=warn event=lock_busy"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
check_once
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
usage >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
Reference in New Issue
Block a user