feat(1c): add company intelligence forecasting layer
This commit is contained in:
@@ -13,6 +13,7 @@
|
|||||||
- `docs/GRAFANA_DASHBOARDS_RU.md` — импорт и сопровождение Grafana dashboard'ов через Ansible API playbook.
|
- `docs/GRAFANA_DASHBOARDS_RU.md` — импорт и сопровождение Grafana dashboard'ов через Ansible API playbook.
|
||||||
- `docs/PRESENTATION_RU.md` — презентационные экраны Grafana и AW-rus со скриншотами.
|
- `docs/PRESENTATION_RU.md` — презентационные экраны Grafana и AW-rus со скриншотами.
|
||||||
- `docs/1C_FILE_ANALYTICS_STACK_RU.md` — production guide по файловой 1С Detmir: topology, rollout, verification, recovery, task principal и hardening.
|
- `docs/1C_FILE_ANALYTICS_STACK_RU.md` — production guide по файловой 1С Detmir: topology, rollout, verification, recovery, task principal и hardening.
|
||||||
|
- `docs/1C_COMPANY_INTELLIGENCE_RU.md` — слой анализа и прогноза по компаниям поверх `clickhouse-1c`: marts, forecasting, API и Grafana.
|
||||||
- `docs/windows/ensemble.md` — orchestration-пакет для Windows-деплоя и проверки.
|
- `docs/windows/ensemble.md` — orchestration-пакет для Windows-деплоя и проверки.
|
||||||
- `docs/linux-client.md` — user-space rollout Linux-клиента ActivityWatch на удалённый `AW server`.
|
- `docs/linux-client.md` — user-space rollout Linux-клиента ActivityWatch на удалённый `AW server`.
|
||||||
- `docs/linux-remote-worker.md` — полный Linux remote-worker stack: GUI, SSH/console и browser admin UI вроде Proxmox `:8006`.
|
- `docs/linux-remote-worker.md` — полный Linux remote-worker stack: GUI, SSH/console и browser admin UI вроде Proxmox `:8006`.
|
||||||
@@ -25,7 +26,7 @@
|
|||||||
- `aw-server/` — установочные скрипты, env-шаблон, systemd unit и RU patch для Web UI.
|
- `aw-server/` — установочные скрипты, env-шаблон, systemd unit и RU patch для Web UI.
|
||||||
- `ansible/` — Ansible-ensemble для автоматизированного сервера (Debian/CT).
|
- `ansible/` — Ansible-ensemble для автоматизированного сервера (Debian/CT).
|
||||||
- `grafana/` — version-controlled Grafana dashboard JSON для RDP/worktime, DLP/ИБ и overview-экранов.
|
- `grafana/` — version-controlled Grafana dashboard JSON для RDP/worktime, DLP/ИБ и overview-экранов.
|
||||||
- `clickhouse-1c/` — отдельный analytics stack для **файловой 1С**: ETL, ClickHouse schema, detections, Grafana catalog и AI Investigator contract.
|
- `clickhouse-1c/` — отдельный analytics stack для **файловой 1С**: ETL, ClickHouse schema, detections, company intelligence marts/forecasting, Grafana catalog и AI Investigator contract.
|
||||||
- `pfsense/` — внешний poller для pfSense API и systemd unit под Debian/Ubuntu utility VM.
|
- `pfsense/` — внешний poller для pfSense API и systemd unit под Debian/Ubuntu utility VM.
|
||||||
- `windows/` — PowerShell toolkit: single-user, domain-users, ensemble orchestration, hardening/recovery, validation, Windows/RDP DLP telemetry (`aw-dlp-incidents_*`, `aw-dlp-endpoint-signals_*`) и session-level presence для удалённых Windows/RDP пользователей (`aw-worktime-sessions_*`).
|
- `windows/` — PowerShell toolkit: single-user, domain-users, ensemble orchestration, hardening/recovery, validation, Windows/RDP DLP telemetry (`aw-dlp-incidents_*`, `aw-dlp-endpoint-signals_*`) и session-level presence для удалённых Windows/RDP пользователей (`aw-worktime-sessions_*`).
|
||||||
- `scripts/quality-gate.sh` — локальный preflight-пайплайн проверок.
|
- `scripts/quality-gate.sh` — локальный preflight-пайплайн проверок.
|
||||||
|
|||||||
@@ -7,3 +7,9 @@ GRAFANA_ADMIN_USER=admin
|
|||||||
GRAFANA_ADMIN_PASSWORD=change-me
|
GRAFANA_ADMIN_PASSWORD=change-me
|
||||||
GRAFANA_PORT=3300
|
GRAFANA_PORT=3300
|
||||||
CLICKHOUSE_HOST=clickhouse
|
CLICKHOUSE_HOST=clickhouse
|
||||||
|
AW_1C_CLICKHOUSE_RUNTIME_HOST=127.0.0.1
|
||||||
|
AW_1C_COMPANY_API_HOST=127.0.0.1
|
||||||
|
AW_1C_COMPANY_API_PORT=8710
|
||||||
|
AW_1C_COMPANY_LOOKBACK_DAYS=30
|
||||||
|
AW_1C_COMPANY_MIN_DAYS=3
|
||||||
|
AW_1C_COMPANY_HORIZONS=7,30
|
||||||
|
|||||||
+22
-2
@@ -31,7 +31,9 @@ File 1C + reglog + host telemetry
|
|||||||
├─ host_events
|
├─ host_events
|
||||||
├─ entity_timeline
|
├─ entity_timeline
|
||||||
├─ detections
|
├─ detections
|
||||||
└─ cases
|
├─ cases
|
||||||
|
├─ company_forecasts
|
||||||
|
└─ company_health_signals
|
||||||
↓
|
↓
|
||||||
Grafana + Alerting + AI Investigator
|
Grafana + Alerting + AI Investigator
|
||||||
```
|
```
|
||||||
@@ -48,11 +50,14 @@ File 1C + reglog + host telemetry
|
|||||||
- `grafana/dashboard-catalog.md` — целевая структура дашбордов.
|
- `grafana/dashboard-catalog.md` — целевая структура дашбордов.
|
||||||
- `grafana/query-pack.sql` — базовые SQL-запросы для панелей.
|
- `grafana/query-pack.sql` — базовые SQL-запросы для панелей.
|
||||||
- `grafana/provisioning/datasources/clickhouse.yml` — provisioned datasource для Grafana.
|
- `grafana/provisioning/datasources/clickhouse.yml` — provisioned datasource для Grafana.
|
||||||
|
- `grafana/provisioning/dashboards/files/1c-company-intelligence.json` — source dashboard для анализа и прогноза по компаниям.
|
||||||
- `detections/build_entity_timeline.sql` — сборка единого timeline слоя.
|
- `detections/build_entity_timeline.sql` — сборка единого timeline слоя.
|
||||||
- `detections/open_cases_from_detections.sql` — шаблон открытия cases из detections.
|
- `detections/open_cases_from_detections.sql` — шаблон открытия cases из detections.
|
||||||
- `ops/etl-cron.example` — пример расписания каждые 6 часов.
|
- `ops/etl-cron.example` — пример расписания каждые 6 часов.
|
||||||
- `ops/retention-policy.md` — минимальная retention policy.
|
- `ops/retention-policy.md` — минимальная retention policy.
|
||||||
- `ai/INVESTIGATOR_API.md` — контракт AI Investigator поверх ClickHouse/cases.
|
- `ai/INVESTIGATOR_API.md` — контракт AI Investigator поверх ClickHouse/cases.
|
||||||
|
- `ai/refresh_company_intelligence.py` — materialization forecast/signals по `counterparty`.
|
||||||
|
- `ai/company_intelligence_api.py` — read-only API для AI/аналитики по компаниям.
|
||||||
|
|
||||||
## Когда использовать именно этот контур
|
## Когда использовать именно этот контур
|
||||||
|
|
||||||
@@ -106,7 +111,20 @@ python etl/load_1c_exports.py --config etl/config.yml
|
|||||||
clickhouse-client --queries-file detections/insert_detections.sql
|
clickhouse-client --queries-file detections/insert_detections.sql
|
||||||
```
|
```
|
||||||
|
|
||||||
6. В Grafana строить dashboards из `grafana/dashboard-catalog.md` и
|
6. Включить company intelligence слой:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clickhouse-client --queries-file clickhouse/init/04_company_intelligence.sql
|
||||||
|
python ai/refresh_company_intelligence.py --host localhost --port 8123 --user default --password change-me --database analytics_1c
|
||||||
|
```
|
||||||
|
|
||||||
|
7. Запустить read-only API:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python ai/company_intelligence_api.py --host 127.0.0.1 --port 8710
|
||||||
|
```
|
||||||
|
|
||||||
|
8. В Grafana строить dashboards из `grafana/dashboard-catalog.md` и
|
||||||
`grafana/query-pack.sql`.
|
`grafana/query-pack.sql`.
|
||||||
|
|
||||||
## Ожидаемые источники данных
|
## Ожидаемые источники данных
|
||||||
@@ -116,6 +134,7 @@ clickhouse-client --queries-file detections/insert_detections.sql
|
|||||||
- журнал регистрации 1С;
|
- журнал регистрации 1С;
|
||||||
- audit/export критичных изменений;
|
- audit/export критичных изменений;
|
||||||
- host telemetry с Windows/RDP host.
|
- host telemetry с Windows/RDP host.
|
||||||
|
- для company intelligence нужны документы с непустым `counterparty`.
|
||||||
|
|
||||||
## Границы
|
## Границы
|
||||||
|
|
||||||
@@ -123,3 +142,4 @@ clickhouse-client --queries-file detections/insert_detections.sql
|
|||||||
- LLM не ходит прямо в production 1С;
|
- LLM не ходит прямо в production 1С;
|
||||||
- в ClickHouse кладутся нормализованные выгрузки и enrichment;
|
- в ClickHouse кладутся нормализованные выгрузки и enrichment;
|
||||||
- case/timeline слой считается вне 1С.
|
- case/timeline слой считается вне 1С.
|
||||||
|
- если `counterparty` в live-выгрузках пустой, company-forecast слой останется корректно пустым.
|
||||||
|
|||||||
@@ -15,6 +15,9 @@ AI Investigator не должен ходить напрямую в файлов
|
|||||||
- что произошло по case `X`;
|
- что произошло по case `X`;
|
||||||
- собрать summary по entity timeline;
|
- собрать summary по entity timeline;
|
||||||
- предложить next steps без write-действий.
|
- предложить next steps без write-действий.
|
||||||
|
- какие компании выпали из активности;
|
||||||
|
- где ожидается спад или рост объёма по компаниям;
|
||||||
|
- какие компании требуют проверки из-за резкого падения документооборота.
|
||||||
|
|
||||||
## Рекомендуемые API endpoints
|
## Рекомендуемые API endpoints
|
||||||
|
|
||||||
@@ -61,6 +64,52 @@ AI Investigator не должен ходить напрямую в файлов
|
|||||||
- related detections
|
- related detections
|
||||||
- related timeline rows
|
- related timeline rows
|
||||||
|
|
||||||
|
### `GET /api/1/analytics-1c/companies/overview`
|
||||||
|
|
||||||
|
Фильтры:
|
||||||
|
|
||||||
|
- `infobase`
|
||||||
|
- `min_signal_score`
|
||||||
|
- `limit`
|
||||||
|
|
||||||
|
Возвращает:
|
||||||
|
|
||||||
|
- компании с активностью за 30 дней
|
||||||
|
- последние сигналы риска
|
||||||
|
- прогноз `amount/docs` на `7/30` дней
|
||||||
|
|
||||||
|
### `GET /api/1/analytics-1c/companies/{counterparty}/summary`
|
||||||
|
|
||||||
|
Возвращает:
|
||||||
|
|
||||||
|
- текущую карточку компании
|
||||||
|
- AI-ready short summary
|
||||||
|
- последние документы
|
||||||
|
- forecasts
|
||||||
|
- signals
|
||||||
|
|
||||||
|
### `GET /api/1/analytics-1c/companies/{counterparty}/forecast`
|
||||||
|
|
||||||
|
Возвращает:
|
||||||
|
|
||||||
|
- `metric`
|
||||||
|
- `horizon_days`
|
||||||
|
- `baseline_daily`
|
||||||
|
- `trend_slope`
|
||||||
|
- `predicted_daily`
|
||||||
|
- `predicted_total`
|
||||||
|
- `confidence`
|
||||||
|
|
||||||
|
### `GET /api/1/analytics-1c/companies/{counterparty}/timeline`
|
||||||
|
|
||||||
|
Возвращает:
|
||||||
|
|
||||||
|
- последние документы по компании
|
||||||
|
- базу
|
||||||
|
- автора
|
||||||
|
- тип операции
|
||||||
|
- статус
|
||||||
|
|
||||||
## Guardrails
|
## Guardrails
|
||||||
|
|
||||||
- read-only SQL;
|
- read-only SQL;
|
||||||
@@ -68,6 +117,7 @@ AI Investigator не должен ходить напрямую в файлов
|
|||||||
- no direct write-back into 1С;
|
- no direct write-back into 1С;
|
||||||
- no direct execution of arbitrary SQL from prompt;
|
- no direct execution of arbitrary SQL from prompt;
|
||||||
- all investigator requests are logged.
|
- all investigator requests are logged.
|
||||||
|
- если live-данные не содержат `counterparty`, company endpoints должны честно возвращать пустой результат, а не симулировать прогноз.
|
||||||
|
|
||||||
## Output style
|
## Output style
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,206 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from decimal import Decimal
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import clickhouse_connect
|
||||||
|
import uvicorn
|
||||||
|
from fastapi import FastAPI, HTTPException, Query
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
p = argparse.ArgumentParser(description="Read-only company intelligence API for analytics_1c")
|
||||||
|
p.add_argument("--host", default=os.getenv("AW_1C_COMPANY_API_HOST", "127.0.0.1"))
|
||||||
|
p.add_argument("--port", type=int, default=int(os.getenv("AW_1C_COMPANY_API_PORT", "8710")))
|
||||||
|
return p.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def q(value: str) -> str:
|
||||||
|
return "'" + value.replace("'", "''") + "'"
|
||||||
|
|
||||||
|
|
||||||
|
def to_plain(value: Any) -> Any:
|
||||||
|
if isinstance(value, Decimal):
|
||||||
|
return float(value)
|
||||||
|
if isinstance(value, datetime):
|
||||||
|
return value.isoformat()
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def rows_to_dict(result) -> list[dict[str, Any]]:
|
||||||
|
return [
|
||||||
|
{name: to_plain(value) for name, value in zip(result.column_names, row)}
|
||||||
|
for row in result.result_rows
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def ch_client():
|
||||||
|
return clickhouse_connect.get_client(
|
||||||
|
host=os.getenv("CLICKHOUSE_HOST", "localhost"),
|
||||||
|
port=int(os.getenv("CLICKHOUSE_PORT", "8123")),
|
||||||
|
username=os.getenv("CLICKHOUSE_USER", "default"),
|
||||||
|
password=os.getenv("CLICKHOUSE_PASSWORD", ""),
|
||||||
|
database=os.getenv("CLICKHOUSE_DB", "analytics_1c"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(title="AW-rus 1C Company Intelligence API", version="1.0.0")
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
def health() -> dict[str, Any]:
|
||||||
|
client = ch_client()
|
||||||
|
summary = rows_to_dict(
|
||||||
|
client.query(
|
||||||
|
"""
|
||||||
|
SELECT
|
||||||
|
countIf(counterparty != '') AS documents_with_counterparty,
|
||||||
|
(SELECT count() FROM analytics_1c.company_forecasts) AS forecasts_total,
|
||||||
|
(SELECT count() FROM analytics_1c.company_health_signals) AS health_signals_total
|
||||||
|
FROM analytics_1c.documents
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
)[0]
|
||||||
|
return {"status": "ok", "generated_at": datetime.now(UTC).isoformat(), **summary}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/1/analytics-1c/companies/overview")
|
||||||
|
def companies_overview(
|
||||||
|
infobase: str | None = None,
|
||||||
|
min_signal_score: int = Query(default=0, ge=0, le=100),
|
||||||
|
limit: int = Query(default=50, ge=1, le=500),
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
client = ch_client()
|
||||||
|
where = [f"signal_score >= {int(min_signal_score)}"]
|
||||||
|
if infobase:
|
||||||
|
where.append(f"infobase = {q(infobase)}")
|
||||||
|
sql = f"""
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
organization,
|
||||||
|
counterparty,
|
||||||
|
last_seen_at,
|
||||||
|
days_since_last_activity,
|
||||||
|
docs_7d,
|
||||||
|
amount_7d,
|
||||||
|
docs_30d,
|
||||||
|
amount_30d,
|
||||||
|
amount_forecast_30d,
|
||||||
|
docs_forecast_30d,
|
||||||
|
signal_severity,
|
||||||
|
signal_score,
|
||||||
|
top_signal
|
||||||
|
FROM analytics_1c.v_company_portfolio_overview
|
||||||
|
WHERE {' AND '.join(where)}
|
||||||
|
ORDER BY signal_score DESC, amount_30d DESC, counterparty
|
||||||
|
LIMIT {int(limit)}
|
||||||
|
"""
|
||||||
|
rows = rows_to_dict(client.query(sql))
|
||||||
|
return {"items": rows, "count": len(rows)}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/1/analytics-1c/companies/{counterparty}/summary")
|
||||||
|
def company_summary(counterparty: str, infobase: str | None = None) -> dict[str, Any]:
|
||||||
|
client = ch_client()
|
||||||
|
filters = [f"counterparty = {q(counterparty)}"]
|
||||||
|
if infobase:
|
||||||
|
filters.append(f"infobase = {q(infobase)}")
|
||||||
|
sql = f"""
|
||||||
|
SELECT *
|
||||||
|
FROM analytics_1c.v_company_portfolio_overview
|
||||||
|
WHERE {' AND '.join(filters)}
|
||||||
|
ORDER BY amount_30d DESC
|
||||||
|
LIMIT 1
|
||||||
|
"""
|
||||||
|
rows = rows_to_dict(client.query(sql))
|
||||||
|
if not rows:
|
||||||
|
raise HTTPException(status_code=404, detail="counterparty not found in analytics_1c.v_company_portfolio_overview")
|
||||||
|
card = rows[0]
|
||||||
|
forecast_sql = f"""
|
||||||
|
SELECT metric, horizon_days, baseline_daily, trend_slope, predicted_daily, predicted_total, confidence, note
|
||||||
|
FROM analytics_1c.v_company_forecasts_current
|
||||||
|
WHERE counterparty = {q(counterparty)}
|
||||||
|
{"AND infobase = " + q(infobase) if infobase else ""}
|
||||||
|
ORDER BY metric, horizon_days
|
||||||
|
"""
|
||||||
|
signals_sql = f"""
|
||||||
|
SELECT generated_at, severity, score, signal_type, summary
|
||||||
|
FROM analytics_1c.v_company_health_current
|
||||||
|
WHERE counterparty = {q(counterparty)}
|
||||||
|
{"AND infobase = " + q(infobase) if infobase else ""}
|
||||||
|
ORDER BY score DESC, generated_at DESC
|
||||||
|
"""
|
||||||
|
timeline_sql = f"""
|
||||||
|
SELECT ts, infobase, doc_type, operation_type, amount, status, author
|
||||||
|
FROM analytics_1c.documents
|
||||||
|
WHERE counterparty = {q(counterparty)}
|
||||||
|
{"AND infobase = " + q(infobase) if infobase else ""}
|
||||||
|
ORDER BY ts DESC
|
||||||
|
LIMIT 20
|
||||||
|
"""
|
||||||
|
forecasts = rows_to_dict(client.query(forecast_sql))
|
||||||
|
signals = rows_to_dict(client.query(signals_sql))
|
||||||
|
timeline = rows_to_dict(client.query(timeline_sql))
|
||||||
|
essence = (
|
||||||
|
f"Компания {counterparty}: за 30 дней документов {card['docs_30d']}, объём {card['amount_30d']}, "
|
||||||
|
f"прогноз на 30 дней {card['amount_forecast_30d']}, риск {card['signal_severity']}."
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"essence": essence,
|
||||||
|
"card": card,
|
||||||
|
"forecasts": forecasts,
|
||||||
|
"signals": signals,
|
||||||
|
"recent_documents": timeline,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/1/analytics-1c/companies/{counterparty}/forecast")
|
||||||
|
def company_forecast(
|
||||||
|
counterparty: str,
|
||||||
|
infobase: str | None = None,
|
||||||
|
horizon_days: int | None = Query(default=None, ge=1, le=365),
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
client = ch_client()
|
||||||
|
filters = [f"counterparty = {q(counterparty)}"]
|
||||||
|
if infobase:
|
||||||
|
filters.append(f"infobase = {q(infobase)}")
|
||||||
|
if horizon_days is not None:
|
||||||
|
filters.append(f"horizon_days = {int(horizon_days)}")
|
||||||
|
sql = f"""
|
||||||
|
SELECT generated_at, infobase, counterparty, metric, horizon_days, baseline_daily, trend_slope, predicted_daily, predicted_total, confidence, note
|
||||||
|
FROM analytics_1c.v_company_forecasts_current
|
||||||
|
WHERE {' AND '.join(filters)}
|
||||||
|
ORDER BY metric, horizon_days
|
||||||
|
"""
|
||||||
|
rows = rows_to_dict(client.query(sql))
|
||||||
|
return {"items": rows, "count": len(rows)}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/1/analytics-1c/companies/{counterparty}/timeline")
|
||||||
|
def company_timeline(
|
||||||
|
counterparty: str,
|
||||||
|
infobase: str | None = None,
|
||||||
|
limit: int = Query(default=100, ge=1, le=500),
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
client = ch_client()
|
||||||
|
filters = [f"counterparty = {q(counterparty)}"]
|
||||||
|
if infobase:
|
||||||
|
filters.append(f"infobase = {q(infobase)}")
|
||||||
|
sql = f"""
|
||||||
|
SELECT ts, infobase, organization, doc_type, doc_number, author, operation_type, amount, status, posted
|
||||||
|
FROM analytics_1c.documents
|
||||||
|
WHERE {' AND '.join(filters)}
|
||||||
|
ORDER BY ts DESC
|
||||||
|
LIMIT {int(limit)}
|
||||||
|
"""
|
||||||
|
rows = rows_to_dict(client.query(sql))
|
||||||
|
return {"items": rows, "count": len(rows)}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
args = parse_args()
|
||||||
|
uvicorn.run(app, host=args.host, port=args.port)
|
||||||
@@ -0,0 +1,301 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
from collections import defaultdict
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, date, datetime, timedelta
|
||||||
|
from statistics import fmean, pstdev
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import clickhouse_connect
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
p = argparse.ArgumentParser(description="Refresh company forecasts and health signals for analytics_1c")
|
||||||
|
p.add_argument("--host", default=os.getenv("CLICKHOUSE_HOST", "localhost"))
|
||||||
|
p.add_argument("--port", type=int, default=int(os.getenv("CLICKHOUSE_PORT", "8123")))
|
||||||
|
p.add_argument("--user", default=os.getenv("CLICKHOUSE_USER", "default"))
|
||||||
|
p.add_argument("--password", default=os.getenv("CLICKHOUSE_PASSWORD", ""))
|
||||||
|
p.add_argument("--database", default=os.getenv("CLICKHOUSE_DB", "analytics_1c"))
|
||||||
|
p.add_argument("--lookback-days", type=int, default=int(os.getenv("AW_1C_COMPANY_LOOKBACK_DAYS", "30")))
|
||||||
|
p.add_argument("--min-days", type=int, default=int(os.getenv("AW_1C_COMPANY_MIN_DAYS", "3")))
|
||||||
|
p.add_argument("--horizons", default=os.getenv("AW_1C_COMPANY_HORIZONS", "7,30"))
|
||||||
|
return p.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class DailyPoint:
|
||||||
|
d: date
|
||||||
|
docs_total: float
|
||||||
|
amount_total: float
|
||||||
|
|
||||||
|
|
||||||
|
def ch_client(args: argparse.Namespace):
|
||||||
|
return clickhouse_connect.get_client(
|
||||||
|
host=args.host,
|
||||||
|
port=args.port,
|
||||||
|
username=args.user,
|
||||||
|
password=args.password,
|
||||||
|
database=args.database,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def query_rows(client, sql: str) -> list[dict[str, Any]]:
|
||||||
|
result = client.query(sql)
|
||||||
|
return [dict(zip(result.column_names, row)) for row in result.result_rows]
|
||||||
|
|
||||||
|
|
||||||
|
def fill_daily_series(points: list[DailyPoint]) -> list[DailyPoint]:
|
||||||
|
if not points:
|
||||||
|
return []
|
||||||
|
by_day = {p.d: p for p in points}
|
||||||
|
current = points[0].d
|
||||||
|
end = points[-1].d
|
||||||
|
filled: list[DailyPoint] = []
|
||||||
|
while current <= end:
|
||||||
|
filled.append(by_day.get(current, DailyPoint(current, 0.0, 0.0)))
|
||||||
|
current += timedelta(days=1)
|
||||||
|
return filled
|
||||||
|
|
||||||
|
|
||||||
|
def linear_slope(values: list[float]) -> float:
|
||||||
|
n = len(values)
|
||||||
|
if n < 2:
|
||||||
|
return 0.0
|
||||||
|
x_mean = (n - 1) / 2
|
||||||
|
y_mean = fmean(values)
|
||||||
|
num = sum((i - x_mean) * (v - y_mean) for i, v in enumerate(values))
|
||||||
|
den = sum((i - x_mean) ** 2 for i in range(n))
|
||||||
|
if den == 0:
|
||||||
|
return 0.0
|
||||||
|
return num / den
|
||||||
|
|
||||||
|
|
||||||
|
def build_forecast(values: list[float], horizon: int, min_days: int, lookback_days: int) -> tuple[float, float, float, float, int, str]:
|
||||||
|
if len(values) < min_days:
|
||||||
|
raise ValueError("not enough data")
|
||||||
|
window = values[-min(len(values), lookback_days):]
|
||||||
|
baseline = fmean(window)
|
||||||
|
slope = linear_slope(window)
|
||||||
|
projected = [max(0.0, baseline + slope * step) for step in range(1, horizon + 1)]
|
||||||
|
predicted_total = sum(projected)
|
||||||
|
predicted_daily = projected[-1] if projected else baseline
|
||||||
|
if len(window) > 1 and baseline > 0:
|
||||||
|
volatility = pstdev(window) / baseline
|
||||||
|
elif len(window) > 1:
|
||||||
|
volatility = pstdev(window)
|
||||||
|
else:
|
||||||
|
volatility = 0.0
|
||||||
|
coverage = min(1.0, len(window) / max(lookback_days, 1))
|
||||||
|
stability = max(0.15, 1.0 - min(volatility, 1.0))
|
||||||
|
confidence = max(0.1, min(0.95, coverage * stability))
|
||||||
|
note_parts: list[str] = []
|
||||||
|
if len(values) < lookback_days:
|
||||||
|
note_parts.append("sparse_history")
|
||||||
|
if abs(slope) < 0.01:
|
||||||
|
note_parts.append("flat_trend")
|
||||||
|
note = ",".join(note_parts) if note_parts else "ok"
|
||||||
|
return baseline, slope, predicted_daily, predicted_total, len(window), note
|
||||||
|
|
||||||
|
|
||||||
|
def severity_score_to_label(score: int) -> str:
|
||||||
|
if score >= 80:
|
||||||
|
return "critical"
|
||||||
|
if score >= 60:
|
||||||
|
return "high"
|
||||||
|
if score >= 35:
|
||||||
|
return "medium"
|
||||||
|
return "low"
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = parse_args()
|
||||||
|
client = ch_client(args)
|
||||||
|
horizons = [int(x.strip()) for x in args.horizons.split(",") if x.strip()]
|
||||||
|
generated_at = datetime.now(UTC).replace(tzinfo=None, microsecond=0)
|
||||||
|
|
||||||
|
daily_rows = query_rows(
|
||||||
|
client,
|
||||||
|
"""
|
||||||
|
SELECT infobase, organization, counterparty, d, docs_total, amount_total
|
||||||
|
FROM analytics_1c.v_counterparty_daily
|
||||||
|
ORDER BY infobase, counterparty, d
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
if not daily_rows:
|
||||||
|
print("no counterparty rows in analytics_1c.v_counterparty_daily; nothing to refresh")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
grouped: dict[tuple[str, str, str], list[DailyPoint]] = defaultdict(list)
|
||||||
|
for row in daily_rows:
|
||||||
|
key = (row["infobase"], row["organization"], row["counterparty"])
|
||||||
|
grouped[key].append(
|
||||||
|
DailyPoint(
|
||||||
|
d=row["d"],
|
||||||
|
docs_total=float(row["docs_total"] or 0),
|
||||||
|
amount_total=float(row["amount_total"] or 0),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
cases_map = {
|
||||||
|
(row["infobase"], row["counterparty"]): int(row["open_cases_total"] or 0)
|
||||||
|
for row in query_rows(
|
||||||
|
client,
|
||||||
|
"""
|
||||||
|
SELECT infobase, entity_id AS counterparty, countIf(status != 'closed') AS open_cases_total
|
||||||
|
FROM analytics_1c.cases
|
||||||
|
WHERE entity_type = 'counterparty'
|
||||||
|
GROUP BY infobase, counterparty
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
detections_map = {
|
||||||
|
(row["infobase"], row["counterparty"]): int(row["detections_total"] or 0)
|
||||||
|
for row in query_rows(
|
||||||
|
client,
|
||||||
|
"""
|
||||||
|
SELECT infobase, entity_id AS counterparty, count() AS detections_total
|
||||||
|
FROM analytics_1c.detections
|
||||||
|
WHERE entity_type = 'counterparty' AND status != 'closed'
|
||||||
|
GROUP BY infobase, counterparty
|
||||||
|
""",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
forecast_rows: list[list[Any]] = []
|
||||||
|
signal_rows: list[list[Any]] = []
|
||||||
|
|
||||||
|
for (infobase, _organization, counterparty), points in grouped.items():
|
||||||
|
points.sort(key=lambda p: p.d)
|
||||||
|
filled = fill_daily_series(points)
|
||||||
|
docs_series = [p.docs_total for p in filled]
|
||||||
|
amount_series = [p.amount_total for p in filled]
|
||||||
|
if len(filled) < args.min_days:
|
||||||
|
continue
|
||||||
|
|
||||||
|
latest_day = filled[-1].d
|
||||||
|
last_7 = filled[-7:]
|
||||||
|
prev_7 = filled[-14:-7]
|
||||||
|
docs_7d = int(sum(p.docs_total for p in last_7))
|
||||||
|
docs_prev_7d = int(sum(p.docs_total for p in prev_7))
|
||||||
|
amount_7d = float(sum(p.amount_total for p in last_7))
|
||||||
|
amount_prev_7d = float(sum(p.amount_total for p in prev_7))
|
||||||
|
days_since_last_activity = (date.today() - latest_day).days
|
||||||
|
open_cases_total = cases_map.get((infobase, counterparty), 0)
|
||||||
|
detections_total = detections_map.get((infobase, counterparty), 0)
|
||||||
|
|
||||||
|
for metric, values in (("docs_total", docs_series), ("amount_total", amount_series)):
|
||||||
|
for horizon in horizons:
|
||||||
|
baseline, slope, predicted_daily, predicted_total, source_days, note = build_forecast(
|
||||||
|
values=values,
|
||||||
|
horizon=horizon,
|
||||||
|
min_days=args.min_days,
|
||||||
|
lookback_days=args.lookback_days,
|
||||||
|
)
|
||||||
|
forecast_rows.append(
|
||||||
|
[
|
||||||
|
generated_at,
|
||||||
|
latest_day,
|
||||||
|
infobase,
|
||||||
|
counterparty,
|
||||||
|
int(horizon),
|
||||||
|
metric,
|
||||||
|
float(baseline),
|
||||||
|
float(slope),
|
||||||
|
float(predicted_daily),
|
||||||
|
float(predicted_total),
|
||||||
|
round(float(max(0.1, min(0.95, 1.0 - abs(slope) / (abs(baseline) + 1.0)))), 4),
|
||||||
|
"linear_baseline",
|
||||||
|
int(source_days),
|
||||||
|
note,
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
signals: list[tuple[str, int, str, str]] = []
|
||||||
|
if days_since_last_activity >= 14 and (docs_prev_7d > 0 or amount_prev_7d > 0):
|
||||||
|
signals.append(("inactive_company", 85, "high", f"Нет активности по компании {counterparty} уже {days_since_last_activity} дн."))
|
||||||
|
if amount_prev_7d > 0 and amount_7d < amount_prev_7d * 0.5:
|
||||||
|
signals.append(("amount_drop", 70, "high", f"Объём по компании {counterparty} упал более чем на 50% неделя к неделе."))
|
||||||
|
if docs_prev_7d > 0 and docs_7d == 0:
|
||||||
|
signals.append(("docs_stopped", 55, "medium", f"По компании {counterparty} прекратился поток документов за последние 7 дней."))
|
||||||
|
if open_cases_total > 0:
|
||||||
|
signals.append(("open_cases", min(95, 40 + open_cases_total * 10), severity_score_to_label(min(95, 40 + open_cases_total * 10)), f"По компании {counterparty} есть открытые кейсы: {open_cases_total}."))
|
||||||
|
if detections_total > 0:
|
||||||
|
signals.append(("open_detections", min(90, 35 + detections_total * 5), severity_score_to_label(min(90, 35 + detections_total * 5)), f"По компании {counterparty} есть активные detections: {detections_total}."))
|
||||||
|
|
||||||
|
for signal_type, score, severity, summary in signals:
|
||||||
|
signal_rows.append(
|
||||||
|
[
|
||||||
|
generated_at,
|
||||||
|
infobase,
|
||||||
|
counterparty,
|
||||||
|
f"{signal_type}:{infobase}:{counterparty}",
|
||||||
|
severity,
|
||||||
|
int(score),
|
||||||
|
signal_type,
|
||||||
|
summary,
|
||||||
|
float(amount_7d),
|
||||||
|
float(amount_prev_7d),
|
||||||
|
int(docs_7d),
|
||||||
|
int(docs_prev_7d),
|
||||||
|
int(max(days_since_last_activity, 0)),
|
||||||
|
int(open_cases_total),
|
||||||
|
int(detections_total),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
if forecast_rows:
|
||||||
|
client.insert(
|
||||||
|
"analytics_1c.company_forecasts",
|
||||||
|
forecast_rows,
|
||||||
|
column_names=[
|
||||||
|
"generated_at",
|
||||||
|
"as_of_date",
|
||||||
|
"infobase",
|
||||||
|
"counterparty",
|
||||||
|
"horizon_days",
|
||||||
|
"metric",
|
||||||
|
"baseline_daily",
|
||||||
|
"trend_slope",
|
||||||
|
"predicted_daily",
|
||||||
|
"predicted_total",
|
||||||
|
"confidence",
|
||||||
|
"model",
|
||||||
|
"source_days",
|
||||||
|
"note",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
if signal_rows:
|
||||||
|
client.insert(
|
||||||
|
"analytics_1c.company_health_signals",
|
||||||
|
signal_rows,
|
||||||
|
column_names=[
|
||||||
|
"generated_at",
|
||||||
|
"infobase",
|
||||||
|
"counterparty",
|
||||||
|
"signal_id",
|
||||||
|
"severity",
|
||||||
|
"score",
|
||||||
|
"signal_type",
|
||||||
|
"summary",
|
||||||
|
"amount_7d",
|
||||||
|
"amount_prev_7d",
|
||||||
|
"docs_7d",
|
||||||
|
"docs_prev_7d",
|
||||||
|
"days_since_last_activity",
|
||||||
|
"open_cases_total",
|
||||||
|
"detections_total",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"company intelligence refreshed: forecasts={len(forecast_rows)} signals={len(signal_rows)} generated_at={generated_at.isoformat()}"
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
fastapi==0.115.2
|
||||||
|
uvicorn[standard]==0.32.0
|
||||||
|
clickhouse-connect==0.7.16
|
||||||
@@ -0,0 +1,201 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS analytics_1c.company_forecasts
|
||||||
|
(
|
||||||
|
generated_at DateTime,
|
||||||
|
as_of_date Date,
|
||||||
|
infobase LowCardinality(String),
|
||||||
|
counterparty String,
|
||||||
|
horizon_days UInt16,
|
||||||
|
metric LowCardinality(String),
|
||||||
|
baseline_daily Float64,
|
||||||
|
trend_slope Float64,
|
||||||
|
predicted_daily Float64,
|
||||||
|
predicted_total Float64,
|
||||||
|
confidence Float32,
|
||||||
|
model LowCardinality(String),
|
||||||
|
source_days UInt16,
|
||||||
|
note String
|
||||||
|
)
|
||||||
|
ENGINE = MergeTree
|
||||||
|
ORDER BY (generated_at, infobase, counterparty, metric, horizon_days);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS analytics_1c.company_health_signals
|
||||||
|
(
|
||||||
|
generated_at DateTime,
|
||||||
|
infobase LowCardinality(String),
|
||||||
|
counterparty String,
|
||||||
|
signal_id String,
|
||||||
|
severity LowCardinality(String),
|
||||||
|
score UInt32,
|
||||||
|
signal_type LowCardinality(String),
|
||||||
|
summary String,
|
||||||
|
amount_7d Float64,
|
||||||
|
amount_prev_7d Float64,
|
||||||
|
docs_7d UInt32,
|
||||||
|
docs_prev_7d UInt32,
|
||||||
|
days_since_last_activity UInt16,
|
||||||
|
open_cases_total UInt32,
|
||||||
|
detections_total UInt32
|
||||||
|
)
|
||||||
|
ENGINE = MergeTree
|
||||||
|
ORDER BY (generated_at, severity, infobase, counterparty, signal_id);
|
||||||
|
|
||||||
|
CREATE VIEW IF NOT EXISTS analytics_1c.v_counterparty_daily AS
|
||||||
|
SELECT
|
||||||
|
toDate(ts) AS d,
|
||||||
|
infobase,
|
||||||
|
organization,
|
||||||
|
counterparty,
|
||||||
|
count() AS docs_total,
|
||||||
|
sum(amount) AS amount_total,
|
||||||
|
countIf(posted = 1) AS posted_docs_total,
|
||||||
|
countIf(posted = 0) AS unposted_docs_total,
|
||||||
|
countIf(status = 'busy') AS busy_docs_total,
|
||||||
|
countIf(status = 'online') AS online_docs_total,
|
||||||
|
uniqExact(doc_type) AS doc_types_total
|
||||||
|
FROM analytics_1c.documents
|
||||||
|
WHERE counterparty != ''
|
||||||
|
GROUP BY d, infobase, organization, counterparty;
|
||||||
|
|
||||||
|
CREATE VIEW IF NOT EXISTS analytics_1c.v_counterparty_latest_activity AS
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
organization,
|
||||||
|
counterparty,
|
||||||
|
max(ts) AS last_seen_at,
|
||||||
|
argMax(doc_type, ts) AS last_doc_type,
|
||||||
|
argMax(operation_type, ts) AS last_operation_type,
|
||||||
|
argMax(status, ts) AS last_status,
|
||||||
|
argMax(amount, ts) AS last_amount,
|
||||||
|
count() AS docs_lifetime,
|
||||||
|
sum(amount) AS amount_lifetime
|
||||||
|
FROM analytics_1c.documents
|
||||||
|
WHERE counterparty != ''
|
||||||
|
GROUP BY infobase, organization, counterparty;
|
||||||
|
|
||||||
|
CREATE VIEW IF NOT EXISTS analytics_1c.v_company_forecasts_current AS
|
||||||
|
SELECT *
|
||||||
|
FROM analytics_1c.company_forecasts
|
||||||
|
WHERE generated_at = (SELECT max(generated_at) FROM analytics_1c.company_forecasts);
|
||||||
|
|
||||||
|
CREATE VIEW IF NOT EXISTS analytics_1c.v_company_health_current AS
|
||||||
|
SELECT *
|
||||||
|
FROM analytics_1c.company_health_signals
|
||||||
|
WHERE generated_at = (SELECT max(generated_at) FROM analytics_1c.company_health_signals);
|
||||||
|
|
||||||
|
CREATE VIEW IF NOT EXISTS analytics_1c.v_company_portfolio_overview AS
|
||||||
|
WITH
|
||||||
|
base AS
|
||||||
|
(
|
||||||
|
SELECT *
|
||||||
|
FROM analytics_1c.v_counterparty_latest_activity
|
||||||
|
),
|
||||||
|
d7 AS
|
||||||
|
(
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
counterparty,
|
||||||
|
sum(docs_total) AS docs_7d,
|
||||||
|
sum(amount_total) AS amount_7d
|
||||||
|
FROM analytics_1c.v_counterparty_daily
|
||||||
|
WHERE d >= today() - 7
|
||||||
|
GROUP BY infobase, counterparty
|
||||||
|
),
|
||||||
|
d30 AS
|
||||||
|
(
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
counterparty,
|
||||||
|
countDistinct(d) AS active_days_30d,
|
||||||
|
sum(docs_total) AS docs_30d,
|
||||||
|
sum(amount_total) AS amount_30d,
|
||||||
|
sum(busy_docs_total) AS busy_docs_30d
|
||||||
|
FROM analytics_1c.v_counterparty_daily
|
||||||
|
WHERE d >= today() - 30
|
||||||
|
GROUP BY infobase, counterparty
|
||||||
|
),
|
||||||
|
signals AS
|
||||||
|
(
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
counterparty,
|
||||||
|
max(score) AS signal_score,
|
||||||
|
argMax(severity, score) AS signal_severity,
|
||||||
|
argMax(summary, score) AS top_signal
|
||||||
|
FROM analytics_1c.v_company_health_current
|
||||||
|
GROUP BY infobase, counterparty
|
||||||
|
),
|
||||||
|
amount_forecast AS
|
||||||
|
(
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
counterparty,
|
||||||
|
predicted_total AS amount_forecast_30d,
|
||||||
|
confidence AS amount_forecast_confidence
|
||||||
|
FROM analytics_1c.v_company_forecasts_current
|
||||||
|
WHERE metric = 'amount_total'
|
||||||
|
AND horizon_days = 30
|
||||||
|
),
|
||||||
|
docs_forecast AS
|
||||||
|
(
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
counterparty,
|
||||||
|
predicted_total AS docs_forecast_30d,
|
||||||
|
confidence AS docs_forecast_confidence
|
||||||
|
FROM analytics_1c.v_company_forecasts_current
|
||||||
|
WHERE metric = 'docs_total'
|
||||||
|
AND horizon_days = 30
|
||||||
|
),
|
||||||
|
cases_current AS
|
||||||
|
(
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
entity_id AS counterparty,
|
||||||
|
countIf(status != 'closed') AS open_cases_total
|
||||||
|
FROM analytics_1c.cases
|
||||||
|
WHERE entity_type = 'counterparty'
|
||||||
|
GROUP BY infobase, counterparty
|
||||||
|
),
|
||||||
|
detections_current AS
|
||||||
|
(
|
||||||
|
SELECT
|
||||||
|
infobase,
|
||||||
|
entity_id AS counterparty,
|
||||||
|
count() AS detections_total
|
||||||
|
FROM analytics_1c.detections
|
||||||
|
WHERE entity_type = 'counterparty'
|
||||||
|
AND status != 'closed'
|
||||||
|
GROUP BY infobase, counterparty
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
base.infobase AS infobase,
|
||||||
|
base.organization,
|
||||||
|
base.counterparty AS counterparty,
|
||||||
|
base.last_seen_at,
|
||||||
|
base.last_doc_type,
|
||||||
|
base.last_operation_type,
|
||||||
|
base.last_status,
|
||||||
|
dateDiff('day', toDate(base.last_seen_at), today()) AS days_since_last_activity,
|
||||||
|
ifNull(d7.docs_7d, 0) AS docs_7d,
|
||||||
|
ifNull(d7.amount_7d, 0) AS amount_7d,
|
||||||
|
ifNull(d30.active_days_30d, 0) AS active_days_30d,
|
||||||
|
ifNull(d30.docs_30d, 0) AS docs_30d,
|
||||||
|
ifNull(d30.amount_30d, 0) AS amount_30d,
|
||||||
|
ifNull(d30.busy_docs_30d, 0) AS busy_docs_30d,
|
||||||
|
ifNull(amount_forecast.amount_forecast_30d, 0) AS amount_forecast_30d,
|
||||||
|
ifNull(amount_forecast.amount_forecast_confidence, 0) AS amount_forecast_confidence,
|
||||||
|
ifNull(docs_forecast.docs_forecast_30d, 0) AS docs_forecast_30d,
|
||||||
|
ifNull(docs_forecast.docs_forecast_confidence, 0) AS docs_forecast_confidence,
|
||||||
|
ifNull(cases_current.open_cases_total, 0) AS open_cases_total,
|
||||||
|
ifNull(detections_current.detections_total, 0) AS detections_total,
|
||||||
|
ifNull(signals.signal_severity, 'none') AS signal_severity,
|
||||||
|
ifNull(signals.signal_score, 0) AS signal_score,
|
||||||
|
ifNull(signals.top_signal, '') AS top_signal
|
||||||
|
FROM base
|
||||||
|
LEFT JOIN d7 ON d7.infobase = base.infobase AND d7.counterparty = base.counterparty
|
||||||
|
LEFT JOIN d30 ON d30.infobase = base.infobase AND d30.counterparty = base.counterparty
|
||||||
|
LEFT JOIN signals ON signals.infobase = base.infobase AND signals.counterparty = base.counterparty
|
||||||
|
LEFT JOIN amount_forecast ON amount_forecast.infobase = base.infobase AND amount_forecast.counterparty = base.counterparty
|
||||||
|
LEFT JOIN docs_forecast ON docs_forecast.infobase = base.infobase AND docs_forecast.counterparty = base.counterparty
|
||||||
|
LEFT JOIN cases_current ON cases_current.infobase = base.infobase AND cases_current.counterparty = base.counterparty
|
||||||
|
LEFT JOIN detections_current ON detections_current.infobase = base.infobase AND detections_current.counterparty = base.counterparty;
|
||||||
@@ -0,0 +1,480 @@
|
|||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 1,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"title": "Активные компании 30д",
|
||||||
|
"type": "stat",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 0,
|
||||||
|
"y": 0,
|
||||||
|
"w": 6,
|
||||||
|
"h": 6
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "table",
|
||||||
|
"rawSql": "SELECT countDistinct(counterparty) AS value FROM analytics_1c.v_counterparty_daily WHERE d >= today() - 30",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"colorMode": "value",
|
||||||
|
"graphMode": "none",
|
||||||
|
"justifyMode": "auto",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "auto"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
},
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"color": "green",
|
||||||
|
"value": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "orange",
|
||||||
|
"value": 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "red",
|
||||||
|
"value": 20
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 2,
|
||||||
|
"title": "Прогноз объёма 30д",
|
||||||
|
"type": "stat",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 6,
|
||||||
|
"y": 0,
|
||||||
|
"w": 6,
|
||||||
|
"h": 6
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "table",
|
||||||
|
"rawSql": "SELECT round(sum(predicted_total), 2) AS value FROM analytics_1c.v_company_forecasts_current WHERE metric = 'amount_total' AND horizon_days = 30",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"colorMode": "value",
|
||||||
|
"graphMode": "none",
|
||||||
|
"justifyMode": "auto",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "auto"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 3,
|
||||||
|
"title": "Компании с high/critical сигналами",
|
||||||
|
"type": "stat",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 12,
|
||||||
|
"y": 0,
|
||||||
|
"w": 6,
|
||||||
|
"h": 6
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "table",
|
||||||
|
"rawSql": "SELECT countDistinct(counterparty) AS value FROM analytics_1c.v_company_health_current WHERE severity IN ('high', 'critical')",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"colorMode": "value",
|
||||||
|
"graphMode": "none",
|
||||||
|
"justifyMode": "auto",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "auto"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 4,
|
||||||
|
"title": "Counterparty cases",
|
||||||
|
"type": "stat",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 18,
|
||||||
|
"y": 0,
|
||||||
|
"w": 6,
|
||||||
|
"h": 6
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "table",
|
||||||
|
"rawSql": "SELECT count() AS value FROM analytics_1c.cases WHERE entity_type = 'counterparty' AND status != 'closed'",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"colorMode": "value",
|
||||||
|
"graphMode": "none",
|
||||||
|
"justifyMode": "auto",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "auto"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 5,
|
||||||
|
"title": "Объём по компаниям",
|
||||||
|
"type": "timeseries",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 0,
|
||||||
|
"y": 6,
|
||||||
|
"w": 12,
|
||||||
|
"h": 8
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "timeSeries",
|
||||||
|
"rawSql": "SELECT toDateTime(d) AS time, sum(amount_total) AS value FROM analytics_1c.v_counterparty_daily WHERE d >= today() - 30 GROUP BY time ORDER BY time",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "list",
|
||||||
|
"placement": "bottom"
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 6,
|
||||||
|
"title": "Количество документов по компаниям",
|
||||||
|
"type": "timeseries",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 12,
|
||||||
|
"y": 6,
|
||||||
|
"w": 12,
|
||||||
|
"h": 8
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "timeSeries",
|
||||||
|
"rawSql": "SELECT toDateTime(d) AS time, sum(docs_total) AS value FROM analytics_1c.v_counterparty_daily WHERE d >= today() - 30 GROUP BY time ORDER BY time",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "list",
|
||||||
|
"placement": "bottom"
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 7,
|
||||||
|
"title": "Портфель компаний",
|
||||||
|
"type": "table",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 0,
|
||||||
|
"y": 14,
|
||||||
|
"w": 16,
|
||||||
|
"h": 10
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "table",
|
||||||
|
"rawSql": "SELECT infobase, counterparty, amount_30d, docs_30d, amount_forecast_30d, signal_severity, signal_score, top_signal, last_seen_at FROM analytics_1c.v_company_portfolio_overview ORDER BY signal_score DESC, amount_30d DESC LIMIT 20",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"showHeader": true,
|
||||||
|
"sortBy": [
|
||||||
|
{
|
||||||
|
"displayName": "signal_score",
|
||||||
|
"desc": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 8,
|
||||||
|
"title": "Последние AI-сигналы",
|
||||||
|
"type": "table",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"gridPos": {
|
||||||
|
"x": 16,
|
||||||
|
"y": 14,
|
||||||
|
"w": 8,
|
||||||
|
"h": 10
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"editorType": "sql",
|
||||||
|
"format": 1,
|
||||||
|
"pluginVersion": "11.2.2",
|
||||||
|
"queryType": "table",
|
||||||
|
"rawSql": "SELECT generated_at, infobase, counterparty, severity, score, signal_type, summary FROM analytics_1c.v_company_health_current ORDER BY score DESC, generated_at DESC LIMIT 20",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"options": {
|
||||||
|
"showHeader": true,
|
||||||
|
"sortBy": [
|
||||||
|
{
|
||||||
|
"displayName": "score",
|
||||||
|
"desc": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {},
|
||||||
|
"overrides": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"refresh": "30s",
|
||||||
|
"schemaVersion": 39,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": [
|
||||||
|
"1c",
|
||||||
|
"companies",
|
||||||
|
"forecast",
|
||||||
|
"clickhouse"
|
||||||
|
],
|
||||||
|
"templating": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"name": "infobase",
|
||||||
|
"label": "Infobase",
|
||||||
|
"type": "query",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"refresh": 1,
|
||||||
|
"definition": "SELECT DISTINCT infobase FROM analytics_1c.v_counterparty_latest_activity ORDER BY infobase",
|
||||||
|
"query": "SELECT DISTINCT infobase FROM analytics_1c.v_counterparty_latest_activity ORDER BY infobase",
|
||||||
|
"multi": true,
|
||||||
|
"includeAll": true,
|
||||||
|
"sort": 1,
|
||||||
|
"current": {
|
||||||
|
"selected": false,
|
||||||
|
"text": "All",
|
||||||
|
"value": "$__all"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "counterparty",
|
||||||
|
"label": "Counterparty",
|
||||||
|
"type": "query",
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana-clickhouse-datasource",
|
||||||
|
"uid": "clickhouse-1c"
|
||||||
|
},
|
||||||
|
"refresh": 1,
|
||||||
|
"definition": "SELECT DISTINCT counterparty FROM analytics_1c.v_counterparty_latest_activity ORDER BY counterparty",
|
||||||
|
"query": "SELECT DISTINCT counterparty FROM analytics_1c.v_counterparty_latest_activity ORDER BY counterparty",
|
||||||
|
"multi": true,
|
||||||
|
"includeAll": true,
|
||||||
|
"sort": 1,
|
||||||
|
"current": {
|
||||||
|
"selected": false,
|
||||||
|
"text": "All",
|
||||||
|
"value": "$__all"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"time": {
|
||||||
|
"from": "now-30d",
|
||||||
|
"to": "now"
|
||||||
|
},
|
||||||
|
"timepicker": {},
|
||||||
|
"timezone": "browser",
|
||||||
|
"title": "1C File - Company Intelligence",
|
||||||
|
"uid": "1c-file-companies",
|
||||||
|
"version": 1,
|
||||||
|
"weekStart": ""
|
||||||
|
}
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=AW-rus 1C Company Intelligence API
|
||||||
|
After=network.target docker.service
|
||||||
|
Requires=docker.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
Environment=AW_1C_ROOT=/opt/activitywatch/clickhouse-1c
|
||||||
|
ExecStart=/opt/activitywatch/clickhouse-1c/ops/run_company_intelligence_api.sh
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -22,3 +22,4 @@ fi
|
|||||||
python3 -m venv "${ROOT}/.venv"
|
python3 -m venv "${ROOT}/.venv"
|
||||||
"${ROOT}/.venv/bin/pip" install --upgrade pip
|
"${ROOT}/.venv/bin/pip" install --upgrade pip
|
||||||
"${ROOT}/.venv/bin/pip" install -r "${ROOT}/etl/requirements.txt"
|
"${ROOT}/.venv/bin/pip" install -r "${ROOT}/etl/requirements.txt"
|
||||||
|
"${ROOT}/.venv/bin/pip" install -r "${ROOT}/ai/requirements.txt"
|
||||||
|
|||||||
@@ -0,0 +1,29 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="${AW_1C_ROOT:-/opt/activitywatch/clickhouse-1c}"
|
||||||
|
ENV_FILE="${ROOT}/.env"
|
||||||
|
VENV="${ROOT}/.venv"
|
||||||
|
|
||||||
|
if [[ ! -f "${ENV_FILE}" ]]; then
|
||||||
|
echo "missing env file: ${ENV_FILE}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ! -x "${VENV}/bin/python" ]]; then
|
||||||
|
echo "missing venv python: ${VENV}/bin/python" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
set -a
|
||||||
|
. "${ENV_FILE}"
|
||||||
|
set +a
|
||||||
|
|
||||||
|
CH_RUNTIME_HOST="${AW_1C_CLICKHOUSE_RUNTIME_HOST:-${CLICKHOUSE_HOST}}"
|
||||||
|
if [[ "${CH_RUNTIME_HOST}" == "clickhouse" ]]; then
|
||||||
|
CH_RUNTIME_HOST="127.0.0.1"
|
||||||
|
fi
|
||||||
|
export CLICKHOUSE_HOST="${CH_RUNTIME_HOST}"
|
||||||
|
|
||||||
|
exec "${VENV}/bin/python" "${ROOT}/ai/company_intelligence_api.py"
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="${AW_1C_ROOT:-/opt/activitywatch/clickhouse-1c}"
|
||||||
|
ENV_FILE="${ROOT}/.env"
|
||||||
|
VENV="${ROOT}/.venv"
|
||||||
|
CH_CONTAINER="${AW_1C_CLICKHOUSE_CONTAINER:-aw-rus-1c-clickhouse}"
|
||||||
|
|
||||||
|
if [[ ! -f "${ENV_FILE}" ]]; then
|
||||||
|
echo "missing env file: ${ENV_FILE}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ! -x "${VENV}/bin/python" ]]; then
|
||||||
|
echo "missing venv python: ${VENV}/bin/python" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! docker ps --format '{{.Names}}' | grep -qx "${CH_CONTAINER}"; then
|
||||||
|
echo "clickhouse container not running: ${CH_CONTAINER}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
. "${ENV_FILE}"
|
||||||
|
|
||||||
|
CH_RUNTIME_HOST="${AW_1C_CLICKHOUSE_RUNTIME_HOST:-${CLICKHOUSE_HOST}}"
|
||||||
|
if [[ "${CH_RUNTIME_HOST}" == "clickhouse" ]]; then
|
||||||
|
CH_RUNTIME_HOST="127.0.0.1"
|
||||||
|
fi
|
||||||
|
|
||||||
|
docker exec -i "${CH_CONTAINER}" clickhouse-client \
|
||||||
|
--user "${CLICKHOUSE_USER}" \
|
||||||
|
--password "${CLICKHOUSE_PASSWORD}" \
|
||||||
|
--database "${CLICKHOUSE_DB}" \
|
||||||
|
< "${ROOT}/clickhouse/init/04_company_intelligence.sql"
|
||||||
|
|
||||||
|
"${VENV}/bin/python" "${ROOT}/ai/refresh_company_intelligence.py" \
|
||||||
|
--host "${CH_RUNTIME_HOST}" \
|
||||||
|
--port "${CLICKHOUSE_PORT}" \
|
||||||
|
--user "${CLICKHOUSE_USER}" \
|
||||||
|
--password "${CLICKHOUSE_PASSWORD}" \
|
||||||
|
--database "${CLICKHOUSE_DB}"
|
||||||
@@ -56,3 +56,11 @@ docker exec -i "${CH_CONTAINER}" clickhouse-client \
|
|||||||
--password "${CLICKHOUSE_PASSWORD}" \
|
--password "${CLICKHOUSE_PASSWORD}" \
|
||||||
--database "${CLICKHOUSE_DB}" \
|
--database "${CLICKHOUSE_DB}" \
|
||||||
< "${ROOT}/detections/open_cases_from_detections.sql"
|
< "${ROOT}/detections/open_cases_from_detections.sql"
|
||||||
|
|
||||||
|
docker exec -i "${CH_CONTAINER}" clickhouse-client \
|
||||||
|
--user "${CLICKHOUSE_USER}" \
|
||||||
|
--password "${CLICKHOUSE_PASSWORD}" \
|
||||||
|
--database "${CLICKHOUSE_DB}" \
|
||||||
|
< "${ROOT}/clickhouse/init/04_company_intelligence.sql"
|
||||||
|
|
||||||
|
"${ROOT}/ops/run_company_intelligence_refresh.sh"
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
# 1C Company Intelligence для AW-rus
|
||||||
|
|
||||||
|
Этот слой строится **поверх** `clickhouse-1c/` и не трогает саму 1С.
|
||||||
|
|
||||||
|
Его задача:
|
||||||
|
|
||||||
|
- анализировать работу с компаниями (`counterparty`);
|
||||||
|
- показывать, где компании выпали из активности;
|
||||||
|
- считать простой, объяснимый прогноз по документам и объёму;
|
||||||
|
- давать read-only API для AI Investigator и внешних аналитических сервисов.
|
||||||
|
|
||||||
|
## Что считается компанией
|
||||||
|
|
||||||
|
В этом контуре компания = `documents.counterparty`.
|
||||||
|
|
||||||
|
Если в live-выгрузках поле `counterparty` пустое, слой остаётся корректно пустым.
|
||||||
|
Он не выдумывает данные и не пытается прогнозировать то, чего нет.
|
||||||
|
|
||||||
|
## Что добавлено
|
||||||
|
|
||||||
|
### ClickHouse
|
||||||
|
|
||||||
|
Файл:
|
||||||
|
|
||||||
|
- `clickhouse-1c/clickhouse/init/04_company_intelligence.sql`
|
||||||
|
|
||||||
|
Создаёт:
|
||||||
|
|
||||||
|
- `company_forecasts`
|
||||||
|
- `company_health_signals`
|
||||||
|
- `v_counterparty_daily`
|
||||||
|
- `v_counterparty_latest_activity`
|
||||||
|
- `v_company_forecasts_current`
|
||||||
|
- `v_company_health_current`
|
||||||
|
- `v_company_portfolio_overview`
|
||||||
|
|
||||||
|
### Forecast refresh
|
||||||
|
|
||||||
|
Файл:
|
||||||
|
|
||||||
|
- `clickhouse-1c/ai/refresh_company_intelligence.py`
|
||||||
|
|
||||||
|
Что делает:
|
||||||
|
|
||||||
|
- строит daily series по `counterparty`;
|
||||||
|
- считает базовую линию и линейный тренд;
|
||||||
|
- материализует прогнозы на `7` и `30` дней;
|
||||||
|
- создаёт health-signals:
|
||||||
|
- `inactive_company`
|
||||||
|
- `amount_drop`
|
||||||
|
- `docs_stopped`
|
||||||
|
- `open_cases`
|
||||||
|
- `open_detections`
|
||||||
|
|
||||||
|
### Read-only API
|
||||||
|
|
||||||
|
Файл:
|
||||||
|
|
||||||
|
- `clickhouse-1c/ai/company_intelligence_api.py`
|
||||||
|
|
||||||
|
Endpoints:
|
||||||
|
|
||||||
|
- `GET /health`
|
||||||
|
- `GET /api/1/analytics-1c/companies/overview`
|
||||||
|
- `GET /api/1/analytics-1c/companies/{counterparty}/summary`
|
||||||
|
- `GET /api/1/analytics-1c/companies/{counterparty}/forecast`
|
||||||
|
- `GET /api/1/analytics-1c/companies/{counterparty}/timeline`
|
||||||
|
|
||||||
|
### Ops
|
||||||
|
|
||||||
|
Файлы:
|
||||||
|
|
||||||
|
- `clickhouse-1c/ops/run_company_intelligence_refresh.sh`
|
||||||
|
- `clickhouse-1c/ops/run_company_intelligence_api.sh`
|
||||||
|
- `clickhouse-1c/ops/aw-1c-company-api.service`
|
||||||
|
|
||||||
|
И `run_ingest_cycle.sh` теперь:
|
||||||
|
|
||||||
|
1. грузит новые выгрузки;
|
||||||
|
2. обновляет timeline/detections/cases;
|
||||||
|
3. применяет `04_company_intelligence.sql`;
|
||||||
|
4. пересчитывает company forecasts/signals.
|
||||||
|
|
||||||
|
## Развёртывание
|
||||||
|
|
||||||
|
### 1. Установить зависимости
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd clickhouse-1c
|
||||||
|
python3 -m venv .venv
|
||||||
|
. .venv/bin/activate
|
||||||
|
pip install -r etl/requirements.txt
|
||||||
|
pip install -r ai/requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Применить schema
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clickhouse-client --queries-file clickhouse/init/04_company_intelligence.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Пересчитать company intelligence
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./ops/run_company_intelligence_refresh.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Запустить API
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./ops/run_company_intelligence_api.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
По умолчанию:
|
||||||
|
|
||||||
|
- host: `127.0.0.1`
|
||||||
|
- port: `8710`
|
||||||
|
|
||||||
|
## Переменные окружения
|
||||||
|
|
||||||
|
См.:
|
||||||
|
|
||||||
|
- `clickhouse-1c/.env.example`
|
||||||
|
|
||||||
|
Ключевые:
|
||||||
|
|
||||||
|
- `AW_1C_COMPANY_API_HOST`
|
||||||
|
- `AW_1C_COMPANY_API_PORT`
|
||||||
|
- `AW_1C_COMPANY_LOOKBACK_DAYS`
|
||||||
|
- `AW_1C_COMPANY_MIN_DAYS`
|
||||||
|
- `AW_1C_COMPANY_HORIZONS`
|
||||||
|
|
||||||
|
## Что прогноз реально означает
|
||||||
|
|
||||||
|
Это не black-box ML и не «магический AI».
|
||||||
|
|
||||||
|
Сейчас используется объяснимый MVP:
|
||||||
|
|
||||||
|
- daily baseline;
|
||||||
|
- linear trend;
|
||||||
|
- confidence;
|
||||||
|
- health signals на простых правилах.
|
||||||
|
|
||||||
|
Этого достаточно для:
|
||||||
|
|
||||||
|
- раннего обнаружения выпадения компаний из потока;
|
||||||
|
- ранжирования портфеля;
|
||||||
|
- AI summary поверх уже объяснимых чисел.
|
||||||
|
|
||||||
|
## Ограничения
|
||||||
|
|
||||||
|
- без `counterparty` в выгрузках слой пустой;
|
||||||
|
- это прогноз тенденции, а не финансовое обещание;
|
||||||
|
- API строго read-only;
|
||||||
|
- никакой записи обратно в 1С нет.
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# 1C Company Intelligence
|
||||||
|
|
||||||
|
Слой `1C Company Intelligence` расширяет `File 1C analytics` и добавляет:
|
||||||
|
|
||||||
|
- mart по `counterparty`;
|
||||||
|
- прогноз по документам и объёму на `7/30` дней;
|
||||||
|
- health-signals по компаниям;
|
||||||
|
- read-only API для AI Investigator;
|
||||||
|
- source dashboard `1c-file-companies`.
|
||||||
|
|
||||||
|
Основной документ:
|
||||||
|
|
||||||
|
- [1C_COMPANY_INTELLIGENCE_RU.md](../1C_COMPANY_INTELLIGENCE_RU.md)
|
||||||
|
|
||||||
|
Ключевая граница:
|
||||||
|
|
||||||
|
- если `counterparty` в live-выгрузках пустой, слой остаётся пустым честно и ничего не симулирует.
|
||||||
@@ -16,6 +16,7 @@
|
|||||||
- [Hayabusa Security Analytics](Hayabusa-Security-Analytics) - текущий production-контур: auto-upload, auto-case, severity scoring и Telegram alerts
|
- [Hayabusa Security Analytics](Hayabusa-Security-Analytics) - текущий production-контур: auto-upload, auto-case, severity scoring и Telegram alerts
|
||||||
- [Security analytics stack v1](../security-analytics-stack-v1.md) - целевая v1-модель без претензии на Splunk-class SIEM
|
- [Security analytics stack v1](../security-analytics-stack-v1.md) - целевая v1-модель без претензии на Splunk-class SIEM
|
||||||
- [File 1C analytics](File-1C-Analytics) - ClickHouse/Grafana/AI Investigator контур для файловой 1С
|
- [File 1C analytics](File-1C-Analytics) - ClickHouse/Grafana/AI Investigator контур для файловой 1С
|
||||||
|
- [1C Company Intelligence](1C-Company-Intelligence) - AI-ready слой анализа и прогноза по компаниям поверх файловой 1С
|
||||||
|
|
||||||
### Компоненты
|
### Компоненты
|
||||||
- [DLP Endpoint Monitoring](DLP-Endpoint-Monitoring) - мониторинг clipboard, печати, USB
|
- [DLP Endpoint Monitoring](DLP-Endpoint-Monitoring) - мониторинг clipboard, печати, USB
|
||||||
|
|||||||
Reference in New Issue
Block a user