feat(rust): advance powershell migration
This commit is contained in:
@@ -45,7 +45,9 @@ File 1C + reglog + host telemetry
|
||||
- `docker-compose.yml` — локальный scaffold ClickHouse + Grafana.
|
||||
- `.env.example` — переменные окружения.
|
||||
- `clickhouse/init/*.sql` — схема БД.
|
||||
- `etl/load_1c_exports.py` — loader CSV/JSON выгрузок в raw/core таблицы.
|
||||
- `adk-rust/crates/aw-1c-ingest` — production writer CSV/JSON выгрузок в
|
||||
raw/core ClickHouse таблицы.
|
||||
- `etl/load_1c_exports.py` — legacy loader для rollback/ручной отладки.
|
||||
- `etl/build_business_event_exports.py` — read-only normalizer из
|
||||
`documents/postings/audit` в canonical `business_events/document_changes`.
|
||||
- `etl/extract_1c_mcp_toolkit.py` — read-only extractor из
|
||||
@@ -64,7 +66,8 @@ File 1C + reglog + host telemetry
|
||||
- `grafana/provisioning/dashboards/files/1c-telemetry-board.json` — telemetry dashboard по состоянию файловых баз, reglog growth, busy markers и host load.
|
||||
- `detections/build_entity_timeline.sql` — сборка единого timeline слоя.
|
||||
- `detections/open_cases_from_detections.sql` — шаблон открытия cases из detections.
|
||||
- `ops/etl-cron.example` — пример расписания каждые 6 часов.
|
||||
- `ops/etl-cron.example` — legacy cron example; production использует
|
||||
`aw-1c-ingest.timer`.
|
||||
- `ops/retention-policy.md` — минимальная retention policy.
|
||||
- `ai/INVESTIGATOR_API.md` — контракт AI Investigator поверх ClickHouse/cases.
|
||||
- `ai/refresh_company_intelligence.py` — materialization forecast/signals по `counterparty`.
|
||||
@@ -110,7 +113,7 @@ mkdir -p landing/{documents,postings,business_events,document_changes,companies,
|
||||
cp etl/config.example.yml etl/config.yml
|
||||
```
|
||||
|
||||
4. Запустить ETL:
|
||||
4. Запустить ingest:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
@@ -119,8 +122,7 @@ pip install -r etl/requirements.txt
|
||||
python etl/extract_1c_mcp_toolkit.py --config etl/config.yml --validate-config
|
||||
python etl/extract_1c_mcp_toolkit.py --config etl/config.yml --dataset documents --dry-run
|
||||
python etl/extract_1c_mcp_toolkit.py --config etl/config.yml
|
||||
python etl/build_business_event_exports.py --config etl/config.yml
|
||||
python etl/load_1c_exports.py --config etl/config.yml
|
||||
/usr/local/bin/aw-1c-ingest-rust --root /opt/activitywatch/clickhouse-1c
|
||||
```
|
||||
|
||||
5. Применить detections:
|
||||
@@ -233,9 +235,7 @@ etl/extract_1c_mcp_toolkit.py
|
||||
↓
|
||||
landing/{documents,postings,business_events,document_changes,companies,reglog}
|
||||
↓
|
||||
etl/build_business_event_exports.py
|
||||
↓
|
||||
etl/load_1c_exports.py
|
||||
aw-1c-ingest-rust
|
||||
```
|
||||
|
||||
Что он умеет:
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=AW-rus file-1C ingest cycle
|
||||
Description=AW-rus file-1C ingest cycle (Rust)
|
||||
After=docker.service
|
||||
Requires=docker.service
|
||||
|
||||
@@ -7,7 +7,7 @@ Requires=docker.service
|
||||
Type=oneshot
|
||||
WorkingDirectory=/opt/activitywatch/clickhouse-1c
|
||||
Environment=AW_1C_ROOT=/opt/activitywatch/clickhouse-1c
|
||||
ExecStart=/opt/activitywatch/clickhouse-1c/ops/run_ingest_cycle.sh
|
||||
ExecStart=/usr/local/bin/aw-1c-ingest-rust --root /opt/activitywatch/clickhouse-1c
|
||||
TimeoutStartSec=45min
|
||||
User=root
|
||||
Group=root
|
||||
|
||||
@@ -1,5 +1,2 @@
|
||||
# every 6 hours: load exports, rebuild timeline, refresh cases
|
||||
0 */6 * * * cd /opt/aw-rus/clickhouse-1c && . .venv/bin/activate && python etl/load_1c_exports.py --config etl/config.yml >> /var/log/aw-rus-1c-etl.log 2>&1
|
||||
10 */6 * * * clickhouse-client --queries-file /opt/aw-rus/clickhouse-1c/detections/build_entity_timeline.sql >> /var/log/aw-rus-1c-timeline.log 2>&1
|
||||
15 */6 * * * clickhouse-client --queries-file /opt/aw-rus/clickhouse-1c/detections/insert_detections.sql >> /var/log/aw-rus-1c-detections.log 2>&1
|
||||
20 */6 * * * clickhouse-client --queries-file /opt/aw-rus/clickhouse-1c/detections/open_cases_from_detections.sql >> /var/log/aw-rus-1c-cases.log 2>&1
|
||||
# Legacy cron fallback only. Production uses aw-1c-ingest.timer.
|
||||
0 * * * * AW_1C_ROOT=/opt/activitywatch/clickhouse-1c /usr/local/bin/aw-1c-ingest-rust --root /opt/activitywatch/clickhouse-1c >> /var/log/aw-rus-1c-ingest.log 2>&1
|
||||
|
||||
Reference in New Issue
Block a user