feat(workforce): add ClickHouse workforce analytics contour
This commit is contained in:
igor04091968
@@ -0,0 +1,5 @@
|
||||
CLICKHOUSE_DB=aw_workforce
|
||||
CLICKHOUSE_HTTP_BIND=127.0.0.1
|
||||
CLICKHOUSE_PORT=8124
|
||||
CLICKHOUSE_NATIVE_BIND=127.0.0.1
|
||||
CLICKHOUSE_NATIVE_PORT=9001
|
||||
@@ -0,0 +1,2 @@
|
||||
.env
|
||||
.local/
|
||||
@@ -0,0 +1,155 @@
|
||||
# ClickHouse Workforce analytics for AWatch-rus / DetMir
|
||||
|
||||
Этот каталог содержит воспроизводимый ClickHouse-слой для привязки событий
|
||||
AWatch-rus к оргструктуре, классификации приложений и доменов, а также для
|
||||
быстрых агрегатов Grafana.
|
||||
|
||||
Слой не заменяет `clickhouse-1c/`. Это отдельный контур для workforce/web
|
||||
аналитики ActivityWatch-событий.
|
||||
|
||||
## Состав
|
||||
|
||||
- `docker-compose.yml` - локальный ClickHouse scaffold.
|
||||
- `clickhouse/init/00_database.sql` - база `aw_workforce`.
|
||||
- `clickhouse/init/01_raw_tables.sql` - нормализованные staging tables для
|
||||
window/browser events.
|
||||
- `clickhouse/init/02_dimensions_dictionaries.sql` - dimension tables и
|
||||
ClickHouse Dictionaries.
|
||||
- `clickhouse/init/03_materialized_views.sql` - агрегированная таблица и
|
||||
materialized views для Grafana.
|
||||
- `clickhouse/init/04_quality_views.sql` - views контроля unknown-зон.
|
||||
- `sample/seed_demo.sql` - минимальные demo-данные для smoke-проверки.
|
||||
- `sample/seed_sharkon2025_p3.sql` - первая реальная привязка
|
||||
`SHARKON2025/sharkon2025/user1/tsj`.
|
||||
- `ops/run_smoke.sh` - локальный smoke для DDL, dictionaries и агрегатов.
|
||||
- `ops/aw-workforce-ingest.service` / `.timer` - production timer для
|
||||
инкрементальной загрузки.
|
||||
- `ops/aw-workforce-ingest.env.example` - переменные окружения loader-а.
|
||||
- `catalog/*.tsv` - управляемые администратором справочники.
|
||||
- `ops/apply_catalogs.sh` - полная загрузка справочников, reload dictionaries,
|
||||
опциональный rebuild агрегатов.
|
||||
- `ops/report_unknowns.sh` - быстрый отчет top unknown users/processes/domains.
|
||||
|
||||
## Быстрый старт
|
||||
|
||||
```bash
|
||||
cd clickhouse-workforce
|
||||
docker compose up -d
|
||||
./ops/run_smoke.sh
|
||||
```
|
||||
|
||||
Локальный scaffold не задает `CLICKHOUSE_USER/PASSWORD` через Docker entrypoint:
|
||||
это оставляет штатный dev-доступ ClickHouse без пароля и не ломает
|
||||
`SOURCE(CLICKHOUSE(...))` у dictionaries. Файл
|
||||
`clickhouse/users.d/99-aw-workforce-local.xml` разрешает HTTP-запросы от Docker
|
||||
host, а HTTP/native порты по умолчанию привязаны только к `127.0.0.1`.
|
||||
|
||||
Скрипт применяет SQL в правильном порядке, загружает demo seed и проверяет:
|
||||
|
||||
- статус dictionaries;
|
||||
- наличие hourly aggregate rows;
|
||||
- daily productivity view;
|
||||
- unknown quality views.
|
||||
|
||||
`sample/seed_demo.sql` добавляет демонстрационные строки. Для чистого повтора
|
||||
локального smoke пересоздайте volume:
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
docker compose up -d
|
||||
./ops/run_smoke.sh
|
||||
```
|
||||
|
||||
## Production порядок
|
||||
|
||||
1. Реальные источники `aw_window_events` и `aw_browser_events` для
|
||||
`SHARKON2025` подтверждены:
|
||||
`docs/clickhouse/AW_WORKFORCE_SOURCES_SHARKON2025_RU.md`.
|
||||
2. Настроить ingest из ActivityWatch/exporter в staging tables.
|
||||
3. Загрузить `dim_workstation_user`, `dim_application_category`,
|
||||
`dim_domain_category`.
|
||||
4. Проверить `system.dictionaries`.
|
||||
5. Включить materialized views.
|
||||
6. Перевести Grafana на `agg_workforce_productivity_hourly` и
|
||||
`v_workforce_productivity_daily`.
|
||||
|
||||
Исправление справочников не пересчитывает старые агрегаты автоматически.
|
||||
Для исторических периодов нужен backfill по регламенту из
|
||||
`docs/clickhouse/DICTIONARIES_IMPLEMENTATION_PLAN_RU.md`.
|
||||
|
||||
## Live ingest P2/P3
|
||||
|
||||
Rust loader находится в `adk-rust/crates/aw-workforce-ingest`.
|
||||
|
||||
Пример загрузки bounded-окна из живого AW API в локальный ClickHouse:
|
||||
|
||||
```bash
|
||||
cargo run --manifest-path ../adk-rust/Cargo.toml -p aw-workforce-ingest -- \
|
||||
--aw-url http://10.10.10.13:5600/api/0 \
|
||||
--clickhouse-url http://127.0.0.1:8124 \
|
||||
--host SHARKON2025 \
|
||||
--hours 24 \
|
||||
--json
|
||||
```
|
||||
|
||||
Применение первой привязки P3:
|
||||
|
||||
```bash
|
||||
docker exec -i aw-rus-workforce-clickhouse clickhouse-client --multiquery \
|
||||
< sample/seed_sharkon2025_p3.sql
|
||||
```
|
||||
|
||||
## Production ingest P4
|
||||
|
||||
В штатном режиме loader запускается без `--since/--until`: он читает
|
||||
`AW_WORKFORCE_STATE_PATH`, берет `last_end - AW_WORKFORCE_OVERLAP_SECONDS`,
|
||||
загружает bounded range и атомарно сохраняет новый `last_end`. Повторная
|
||||
загрузка overlap-окна не удваивает данные, потому что loader перед вставкой
|
||||
проверяет `source_bucket + source_event_id`.
|
||||
|
||||
Runtime-файлы:
|
||||
|
||||
```bash
|
||||
cd clickhouse-workforce
|
||||
sudo bash ./ops/bootstrap_runtime.sh
|
||||
sudo install -m 0755 ../adk-rust/target/release/aw-workforce-ingest \
|
||||
/usr/local/bin/aw-workforce-ingest
|
||||
sudo editor /etc/activitywatch/aw-workforce-ingest.env
|
||||
sudo systemctl enable --now aw-workforce-ingest.timer
|
||||
```
|
||||
|
||||
Ручная production-проверка одного цикла:
|
||||
|
||||
```bash
|
||||
sudo systemctl start aw-workforce-ingest.service
|
||||
sudo journalctl -u aw-workforce-ingest.service -n 80 --no-pager
|
||||
```
|
||||
|
||||
## Admin workflow справочников P5
|
||||
|
||||
Справочники ведутся через `catalog/*.tsv`. Это полный source of truth:
|
||||
`ops/apply_catalogs.sh` очищает dimension tables, загружает TSV, reload-ит
|
||||
dictionaries и, если нужно, пересобирает агрегаты.
|
||||
|
||||
Посмотреть слепые зоны:
|
||||
|
||||
```bash
|
||||
./ops/report_unknowns.sh
|
||||
```
|
||||
|
||||
Добавить или изменить категорию:
|
||||
|
||||
```bash
|
||||
editor catalog/application_categories.tsv
|
||||
REBUILD_AGGREGATES=1 ./ops/apply_catalogs.sh
|
||||
```
|
||||
|
||||
Убрать запись из отчетов без потери аудита: поставить `is_active=0` в TSV и
|
||||
запустить:
|
||||
|
||||
```bash
|
||||
REBUILD_AGGREGATES=1 ./ops/apply_catalogs.sh
|
||||
```
|
||||
|
||||
Если менялись только future-facing справочники и старые агрегаты пересчитывать
|
||||
не нужно, можно запустить без `REBUILD_AGGREGATES=1`.
|
||||
@@ -0,0 +1,118 @@
|
||||
TRUNCATE TABLE aw_workforce.agg_workforce_productivity_hourly;
|
||||
|
||||
INSERT INTO aw_workforce.agg_workforce_productivity_hourly
|
||||
SELECT
|
||||
toStartOfHour(event_time) AS bucket_start,
|
||||
toDate(event_time) AS event_date,
|
||||
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS branch,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS department,
|
||||
|
||||
'desktop' AS activity_type,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_application_category', 'is_active', process_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_application_category', 'category', process_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_application_category', 'category', process_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS category,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_application_category', 'is_active', process_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_application_category', 'productivity_class', process_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_application_category', 'productivity_class', process_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS productivity_class,
|
||||
|
||||
toUInt64(sum(duration_sec)) AS duration_sec,
|
||||
toUInt64(count()) AS event_count,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'employee_name', (host_name, user_login), '') != '',
|
||||
0,
|
||||
1
|
||||
))) AS unknown_subject_events,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_application_category', 'is_active', process_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_application_category', 'category', process_name, '') NOT IN ('', 'unknown'),
|
||||
0,
|
||||
1
|
||||
))) AS unknown_category_events
|
||||
FROM aw_workforce.aw_window_events
|
||||
GROUP BY
|
||||
bucket_start,
|
||||
event_date,
|
||||
branch,
|
||||
department,
|
||||
activity_type,
|
||||
category,
|
||||
productivity_class;
|
||||
|
||||
INSERT INTO aw_workforce.agg_workforce_productivity_hourly
|
||||
WITH
|
||||
lowerUTF8(
|
||||
domain(if(position(url, '://') = 0, concat('http://', url), url))
|
||||
) AS domain_name
|
||||
SELECT
|
||||
toStartOfHour(event_time) AS bucket_start,
|
||||
toDate(event_time) AS event_date,
|
||||
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS branch,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS department,
|
||||
|
||||
'browser' AS activity_type,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_domain_category', 'is_active', domain_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_domain_category', 'category', domain_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_domain_category', 'category', domain_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS category,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_domain_category', 'is_active', domain_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_domain_category', 'productivity_class', domain_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_domain_category', 'productivity_class', domain_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS productivity_class,
|
||||
|
||||
toUInt64(sum(duration_sec)) AS duration_sec,
|
||||
toUInt64(count()) AS event_count,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'employee_name', (host_name, user_login), '') != '',
|
||||
0,
|
||||
1
|
||||
))) AS unknown_subject_events,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_domain_category', 'is_active', domain_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_domain_category', 'category', domain_name, '') NOT IN ('', 'unknown'),
|
||||
0,
|
||||
1
|
||||
))) AS unknown_category_events
|
||||
FROM aw_workforce.aw_browser_events
|
||||
WHERE domain_name != ''
|
||||
GROUP BY
|
||||
bucket_start,
|
||||
event_date,
|
||||
branch,
|
||||
department,
|
||||
activity_type,
|
||||
category,
|
||||
productivity_class;
|
||||
@@ -0,0 +1,43 @@
|
||||
# Workforce catalogs
|
||||
|
||||
Эти TSV-файлы являются source of truth для справочников `aw_workforce`.
|
||||
|
||||
## Операции администратора
|
||||
|
||||
- Добавить категорию: добавить строку в соответствующий `*.tsv`, поставить
|
||||
`is_active=1`, запустить `ops/apply_catalogs.sh`.
|
||||
- Изменить категорию: изменить строку в `*.tsv`, запустить
|
||||
`REBUILD_AGGREGATES=1 ops/apply_catalogs.sh`.
|
||||
- Удалить категорию из отчетов: либо удалить строку из `*.tsv`, либо оставить
|
||||
строку для аудита и поставить `is_active=0`, затем запустить
|
||||
`REBUILD_AGGREGATES=1 ops/apply_catalogs.sh`.
|
||||
|
||||
`is_active=0` трактуется отчетами как `unknown`: запись остается видимой в
|
||||
каталоге, но не используется для обогащения.
|
||||
|
||||
## Файлы
|
||||
|
||||
- `workstation_users.tsv` - привязка `host_name + user_login` к оргструктуре.
|
||||
- `application_categories.tsv` - классификация desktop processes.
|
||||
- `domain_categories.tsv` - классификация browser domains.
|
||||
|
||||
Формат: `TabSeparatedWithNames`, первая строка - имена колонок. Не используйте
|
||||
tab-символы внутри значений.
|
||||
|
||||
## Таксономия РФ baseline
|
||||
|
||||
Baseline `catalog-ru-20260623` делит домены и приложения на рабочие для РФ
|
||||
категории: `1c`, `edo_reporting`, `reporting`, `banking`, `government`,
|
||||
`procurement`, `business_reference`, `legal_reference`, `mail`, `office`,
|
||||
`pdf`, `browser`, `cloud_docs`, `communication`, `developer`, `admin_tool`,
|
||||
`remote_admin`, `security_crypto`, `search`, `maps_reference`, `marketplace`,
|
||||
`news`, `social`, `media`, `gaming`, `system`.
|
||||
|
||||
Правило ведения: справочник хранит только точные ключи. Для приложений это
|
||||
нормализованный `process_name` в нижнем регистре, например `1cv8c.exe`; для
|
||||
web - точный host из URL, например `online.sbis.ru`. Wildcard-строки не
|
||||
используются, потому что ClickHouse Dictionary выполняет точный lookup.
|
||||
|
||||
`productivity_class` держите в одном из значений: `productive`, `neutral`,
|
||||
`non_productive`, `unknown`. Для облаков, мессенджеров, AI и внешней почты
|
||||
ставьте `risk_level=medium`, если нужна последующая DLP/policy проверка.
|
||||
@@ -0,0 +1,97 @@
|
||||
process_name application_name vendor category productivity_class risk_level is_system source comment is_active
|
||||
1cv8.exe 1C:Enterprise 1C 1c productive low 0 catalog-ru-20260623 1C thick client 1
|
||||
1cv8c.exe 1C:Enterprise client 1C 1c productive low 0 catalog-ru-20260623 1C current RDP workload 1
|
||||
1cv8s.exe 1C:Enterprise starter 1C 1c productive low 0 catalog-ru-20260623 1C server or launcher process 1
|
||||
1cestart.exe 1C:Enterprise launcher 1C 1c productive low 0 catalog-ru-20260623 1C launcher 1
|
||||
1cv8u.exe 1C:Enterprise updater 1C 1c productive low 0 catalog-ru-20260623 1C updater 1
|
||||
sbis.exe SBIS SBIS edo_reporting productive low 0 catalog-ru-20260623 EDO and reporting client 1
|
||||
sbisplugin.exe SBIS Plugin SBIS edo_reporting productive low 0 catalog-ru-20260623 SBIS browser integration 1
|
||||
saby.exe Saby SBIS edo_reporting productive low 0 catalog-ru-20260623 Saby business client 1
|
||||
sabyapps.exe Saby Apps SBIS edo_reporting productive low 0 catalog-ru-20260623 Saby desktop apps 1
|
||||
diadoc.exe Diadoc Kontur edo_reporting productive low 0 catalog-ru-20260623 Electronic document exchange client 1
|
||||
kontur.plugin.exe Kontur Plugin Kontur edo_reporting productive low 0 catalog-ru-20260623 Kontur browser integration 1
|
||||
extern.exe Kontur Extern Kontur edo_reporting productive low 0 catalog-ru-20260623 Electronic reporting client 1
|
||||
cryptoarm.exe CryptoARM Digital Technologies security_crypto productive low 0 catalog-ru-20260623 Crypto signing tool 1
|
||||
cryptcp.exe CryptoPro CryptCP CryptoPro security_crypto productive low 0 catalog-ru-20260623 CryptoPro command line signing tool 1
|
||||
cprocsp.exe CryptoPro CSP CryptoPro security_crypto productive low 0 catalog-ru-20260623 CryptoPro CSP utility 1
|
||||
certmgr.exe Certificate Manager Microsoft security_crypto productive low 0 catalog-ru-20260623 Certificate management 1
|
||||
cadesplugin.exe CAdES Plugin CryptoPro security_crypto productive low 0 catalog-ru-20260623 Browser crypto plugin 1
|
||||
chrome.exe Google Chrome Google browser neutral low 0 catalog-ru-20260623 Browser; domains classify productivity 1
|
||||
msedge.exe Microsoft Edge Microsoft browser neutral low 0 catalog-ru-20260623 Browser; domains classify productivity 1
|
||||
browser.exe Yandex Browser Yandex browser neutral low 0 catalog-ru-20260623 Browser; domains classify productivity 1
|
||||
firefox.exe Mozilla Firefox Mozilla browser neutral low 0 catalog-ru-20260623 Browser; domains classify productivity 1
|
||||
opera.exe Opera Opera browser neutral low 0 catalog-ru-20260623 Browser; domains classify productivity 1
|
||||
iexplore.exe Internet Explorer Microsoft browser neutral medium 0 catalog-ru-20260623 Legacy browser; domains classify productivity 1
|
||||
outlook.exe Microsoft Outlook Microsoft mail productive low 0 catalog-ru-20260623 Business mail client 1
|
||||
thunderbird.exe Thunderbird Mozilla mail productive low 0 catalog-ru-20260623 Mail client 1
|
||||
winword.exe Microsoft Word Microsoft office productive low 0 catalog-ru-20260623 Office document editor 1
|
||||
excel.exe Microsoft Excel Microsoft office productive low 0 catalog-ru-20260623 Spreadsheet editor 1
|
||||
powerpnt.exe Microsoft PowerPoint Microsoft office productive low 0 catalog-ru-20260623 Presentation editor 1
|
||||
onenote.exe Microsoft OneNote Microsoft office productive low 0 catalog-ru-20260623 Notes and documents 1
|
||||
msaccess.exe Microsoft Access Microsoft office productive low 0 catalog-ru-20260623 Database office tool 1
|
||||
soffice.bin LibreOffice The Document Foundation office productive low 0 catalog-ru-20260623 Office suite 1
|
||||
soffice.exe LibreOffice The Document Foundation office productive low 0 catalog-ru-20260623 Office suite launcher 1
|
||||
swriter.exe LibreOffice Writer The Document Foundation office productive low 0 catalog-ru-20260623 Office document editor 1
|
||||
scalc.exe LibreOffice Calc The Document Foundation office productive low 0 catalog-ru-20260623 Spreadsheet editor 1
|
||||
simpress.exe LibreOffice Impress The Document Foundation office productive low 0 catalog-ru-20260623 Presentation editor 1
|
||||
notepad.exe Notepad Microsoft office neutral low 0 catalog-ru-20260623 Text editor 1
|
||||
notepad++.exe Notepad++ Notepad++ office productive low 0 catalog-ru-20260623 Text and config editor 1
|
||||
acrord32.exe Adobe Acrobat Reader Adobe pdf productive low 0 catalog-ru-20260623 PDF reader 1
|
||||
acrobat.exe Adobe Acrobat Adobe pdf productive low 0 catalog-ru-20260623 PDF editor 1
|
||||
foxitpdfreader.exe Foxit PDF Reader Foxit pdf productive low 0 catalog-ru-20260623 PDF reader 1
|
||||
pdfxedit.exe PDF-XChange Editor Tracker Software pdf productive low 0 catalog-ru-20260623 PDF editor 1
|
||||
7zfm.exe 7-Zip File Manager 7-Zip archive productive low 0 catalog-ru-20260623 Archive manager 1
|
||||
7zg.exe 7-Zip GUI 7-Zip archive productive low 0 catalog-ru-20260623 Archive tool 1
|
||||
7z.exe 7-Zip CLI 7-Zip archive productive low 0 catalog-ru-20260623 Archive command line tool 1
|
||||
winrar.exe WinRAR RARLAB archive productive low 0 catalog-ru-20260623 Archive manager 1
|
||||
explorer.exe Windows Explorer Microsoft system neutral low 1 catalog-ru-20260623 Windows shell and file manager 1
|
||||
dwm.exe Desktop Window Manager Microsoft system neutral low 1 catalog-ru-20260623 Windows desktop compositor 1
|
||||
taskhostw.exe Task Host Microsoft system neutral low 1 catalog-ru-20260623 Windows task host 1
|
||||
applicationframehost.exe Application Frame Host Microsoft system neutral low 1 catalog-ru-20260623 Windows app frame host 1
|
||||
startmenuexperiencehost.exe Start Menu Experience Host Microsoft system neutral low 1 catalog-ru-20260623 Windows Start menu 1
|
||||
searchapp.exe Windows Search Microsoft system neutral low 1 catalog-ru-20260623 Windows search UI 1
|
||||
systemsettings.exe Windows Settings Microsoft system neutral low 1 catalog-ru-20260623 Windows settings 1
|
||||
rdpclip.exe RDP Clipboard Microsoft system neutral low 1 catalog-ru-20260623 RDP clipboard process 1
|
||||
conhost.exe Console Host Microsoft system neutral low 1 catalog-ru-20260623 Windows console host 1
|
||||
taskmgr.exe Task Manager Microsoft admin_tool productive low 1 catalog-ru-20260623 Administrative diagnostics 1
|
||||
mmc.exe Microsoft Management Console Microsoft admin_tool productive low 1 catalog-ru-20260623 Administrative console 1
|
||||
eventvwr.exe Event Viewer Microsoft admin_tool productive low 1 catalog-ru-20260623 Administrative diagnostics 1
|
||||
services.exe Windows Services Microsoft admin_tool productive low 1 catalog-ru-20260623 Service control manager 1
|
||||
compmgmtlauncher.exe Computer Management Microsoft admin_tool productive low 1 catalog-ru-20260623 Administrative console launcher 1
|
||||
regedit.exe Registry Editor Microsoft admin_tool productive medium 1 catalog-ru-20260623 Registry editor 1
|
||||
cmd.exe Command Prompt Microsoft admin_tool productive medium 1 catalog-ru-20260623 Command shell 1
|
||||
powershell.exe Windows PowerShell Microsoft admin_tool productive medium 1 catalog-ru-20260623 Administrative shell 1
|
||||
pwsh.exe PowerShell Microsoft admin_tool productive medium 1 catalog-ru-20260623 Administrative shell 1
|
||||
windowsterminal.exe Windows Terminal Microsoft admin_tool productive medium 0 catalog-ru-20260623 Administrative terminal 1
|
||||
wt.exe Windows Terminal Microsoft admin_tool productive medium 0 catalog-ru-20260623 Administrative terminal launcher 1
|
||||
mstsc.exe Remote Desktop Connection Microsoft remote_admin productive low 0 catalog-ru-20260623 RDP client 1
|
||||
putty.exe PuTTY PuTTY remote_admin productive low 0 catalog-ru-20260623 SSH client 1
|
||||
winscp.exe WinSCP WinSCP remote_admin productive low 0 catalog-ru-20260623 SFTP and SCP client 1
|
||||
filezilla.exe FileZilla FileZilla remote_admin productive low 0 catalog-ru-20260623 File transfer client 1
|
||||
procexp.exe Process Explorer Microsoft admin_tool productive medium 0 catalog-ru-20260623 Advanced diagnostics 1
|
||||
procexp64.exe Process Explorer Microsoft admin_tool productive medium 0 catalog-ru-20260623 Advanced diagnostics 1
|
||||
procmon.exe Process Monitor Microsoft admin_tool productive medium 0 catalog-ru-20260623 Advanced diagnostics 1
|
||||
procmon64.exe Process Monitor Microsoft admin_tool productive medium 0 catalog-ru-20260623 Advanced diagnostics 1
|
||||
telegram.exe Telegram Telegram communication neutral medium 0 catalog-ru-20260623 Messenger; business use depends on policy 1
|
||||
whatsapp.exe WhatsApp Meta communication neutral medium 0 catalog-ru-20260623 Messenger; business use depends on policy 1
|
||||
teams.exe Microsoft Teams Microsoft communication productive low 0 catalog-ru-20260623 Business communication 1
|
||||
ms-teams.exe Microsoft Teams Microsoft communication productive low 0 catalog-ru-20260623 Business communication 1
|
||||
zoom.exe Zoom Zoom communication productive low 0 catalog-ru-20260623 Business communication 1
|
||||
skype.exe Skype Microsoft communication neutral medium 0 catalog-ru-20260623 Messenger 1
|
||||
discord.exe Discord Discord social non_productive medium 0 catalog-ru-20260623 Social and gaming communication 1
|
||||
yandexdisk.exe Yandex Disk Yandex cloud_docs productive medium 0 catalog-ru-20260623 Cloud storage; data handling policy applies 1
|
||||
yandexdisk2.exe Yandex Disk Yandex cloud_docs productive medium 0 catalog-ru-20260623 Cloud storage; data handling policy applies 1
|
||||
onedrive.exe OneDrive Microsoft cloud_docs productive medium 0 catalog-ru-20260623 Cloud storage; data handling policy applies 1
|
||||
dropbox.exe Dropbox Dropbox cloud_docs neutral medium 0 catalog-ru-20260623 Cloud storage; data handling policy applies 1
|
||||
code.exe Visual Studio Code Microsoft developer productive low 0 catalog-ru-20260623 Code editor and admin scripting 1
|
||||
devenv.exe Visual Studio Microsoft developer productive low 0 catalog-ru-20260623 Development IDE 1
|
||||
git.exe Git Git developer productive low 0 catalog-ru-20260623 Version control CLI 1
|
||||
docker desktop.exe Docker Desktop Docker developer productive low 0 catalog-ru-20260623 Container development tool 1
|
||||
vlc.exe VLC VideoLAN media non_productive medium 0 catalog-ru-20260623 Media player 1
|
||||
wmplayer.exe Windows Media Player Microsoft media non_productive medium 0 catalog-ru-20260623 Media player 1
|
||||
spotify.exe Spotify Spotify media non_productive medium 0 catalog-ru-20260623 Music streaming 1
|
||||
steam.exe Steam Valve gaming non_productive high 0 catalog-ru-20260623 Gaming platform 1
|
||||
gamebar.exe Xbox Game Bar Microsoft gaming non_productive high 0 catalog-ru-20260623 Gaming overlay 1
|
||||
securityhealthsystray.exe Windows Security Microsoft security_crypto neutral low 1 catalog-ru-20260623 Windows security UI 1
|
||||
avgui.exe AVG Antivirus AVG security_crypto neutral low 0 catalog-ru-20260623 Endpoint security UI 1
|
||||
avpui.exe Kaspersky Kaspersky security_crypto neutral low 0 catalog-ru-20260623 Endpoint security UI 1
|
||||
|
@@ -0,0 +1,128 @@
|
||||
domain site_name category productivity_class risk_level business_allowed source comment is_active
|
||||
intranet.local Internal portal internal_service productive low 1 catalog-ru-20260623 Internal work portal 1
|
||||
dbo.sevnb.ru SEVNB online banking banking productive low 1 catalog-ru-20260623 Banking workflow seen in RDP title stream 1
|
||||
sberbank.ru Sberbank banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
online.sberbank.ru Sberbank Online banking productive low 1 catalog-ru-20260623 Russian banking web client 1
|
||||
sbi.sberbank.ru SberBusiness banking productive low 1 catalog-ru-20260623 Sber business banking 1
|
||||
business-online.sberbank.ru SberBusiness banking productive low 1 catalog-ru-20260623 Sber business banking legacy host 1
|
||||
vtb.ru VTB banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
dbo.vtb.ru VTB Business Online banking productive low 1 catalog-ru-20260623 VTB business banking 1
|
||||
alfabank.ru Alfa-Bank banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
ibank.alfabank.ru Alfa-Bank Business banking productive low 1 catalog-ru-20260623 Alfa business banking 1
|
||||
tbank.ru T-Bank banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
business.tbank.ru T-Business banking productive low 1 catalog-ru-20260623 T-Bank business banking 1
|
||||
tinkoff.ru Tinkoff banking productive low 1 catalog-ru-20260623 T-Bank legacy domain 1
|
||||
business.tinkoff.ru Tinkoff Business banking productive low 1 catalog-ru-20260623 T-Bank business legacy domain 1
|
||||
tochka.com Tochka Bank banking productive low 1 catalog-ru-20260623 Business banking 1
|
||||
bspb.ru Bank Saint Petersburg banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
dbo.bspb.ru Bank Saint Petersburg Business banking productive low 1 catalog-ru-20260623 Business banking 1
|
||||
gazprombank.ru Gazprombank banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
bankuralsib.ru Uralsib banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
rshb.ru Russian Agricultural Bank banking productive low 1 catalog-ru-20260623 Russian banking and business services 1
|
||||
modulbank.ru Modulbank banking productive low 1 catalog-ru-20260623 Business banking 1
|
||||
gosuslugi.ru Gosuslugi government productive low 1 catalog-ru-20260623 Russian government services 1
|
||||
esia.gosuslugi.ru ESIA government productive low 1 catalog-ru-20260623 Government authentication 1
|
||||
lk.gosuslugi.ru Gosuslugi account government productive low 1 catalog-ru-20260623 Government services account 1
|
||||
nalog.gov.ru FNS government productive low 1 catalog-ru-20260623 Russian tax service 1
|
||||
service.nalog.ru FNS services government productive low 1 catalog-ru-20260623 Russian tax service portals 1
|
||||
egrul.nalog.ru FNS EGRUL government productive low 1 catalog-ru-20260623 Company registry lookup 1
|
||||
pb.nalog.ru FNS transparent business government productive low 1 catalog-ru-20260623 Tax counterparty lookup 1
|
||||
lkfl2.nalog.ru FNS personal account government productive low 1 catalog-ru-20260623 Tax account 1
|
||||
lkul.nalog.ru FNS legal entity account government productive low 1 catalog-ru-20260623 Tax account for legal entities 1
|
||||
fssp.gov.ru FSSP government productive low 1 catalog-ru-20260623 Bailiff service lookup 1
|
||||
rosreestr.gov.ru Rosreestr government productive low 1 catalog-ru-20260623 Real estate registry 1
|
||||
kad.arbitr.ru Arbitration cases government productive low 1 catalog-ru-20260623 Arbitration case database 1
|
||||
my.arbitr.ru Arbitration e-filing government productive low 1 catalog-ru-20260623 Arbitration electronic filing 1
|
||||
sudrf.ru Courts of Russia government productive low 1 catalog-ru-20260623 Court information 1
|
||||
zakupki.gov.ru Unified procurement government productive low 1 catalog-ru-20260623 State procurement 1
|
||||
torgi.gov.ru Torgi.gov government productive low 1 catalog-ru-20260623 State auctions and property tenders 1
|
||||
mos.ru Mos.ru government productive low 1 catalog-ru-20260623 Moscow city services 1
|
||||
cbr.ru Bank of Russia government productive low 1 catalog-ru-20260623 Central bank reference data 1
|
||||
rosstat.gov.ru Rosstat government productive low 1 catalog-ru-20260623 Statistics and reporting 1
|
||||
sfr.gov.ru Social Fund government productive low 1 catalog-ru-20260623 Social fund services 1
|
||||
trudvsem.ru Trudvsem government productive low 1 catalog-ru-20260623 Labor and vacancies portal 1
|
||||
kontur.ru Kontur edo_reporting productive low 1 catalog-ru-20260623 EDO reporting and business checks 1
|
||||
extern.kontur.ru Kontur Extern edo_reporting productive low 1 catalog-ru-20260623 Electronic reporting 1
|
||||
diadoc.kontur.ru Diadoc edo_reporting productive low 1 catalog-ru-20260623 Electronic document exchange 1
|
||||
focus.kontur.ru Kontur Focus business_reference productive low 1 catalog-ru-20260623 Counterparty checks 1
|
||||
normativ.kontur.ru Kontur Normativ business_reference productive low 1 catalog-ru-20260623 Legal and accounting reference 1
|
||||
sbis.ru SBIS edo_reporting productive low 1 catalog-ru-20260623 EDO reporting and business services 1
|
||||
online.sbis.ru SBIS Online edo_reporting productive low 1 catalog-ru-20260623 EDO reporting portal 1
|
||||
saby.ru Saby edo_reporting productive low 1 catalog-ru-20260623 SBIS/Saby services 1
|
||||
ofd.ru OFD reporting productive low 1 catalog-ru-20260623 Fiscal data operator 1
|
||||
ofd.yandex.ru Yandex OFD reporting productive low 1 catalog-ru-20260623 Fiscal data operator 1
|
||||
1-ofd.ru 1-OFD reporting productive low 1 catalog-ru-20260623 Fiscal data operator 1
|
||||
sberbank-ast.ru Sberbank AST procurement productive low 1 catalog-ru-20260623 Electronic trading platform 1
|
||||
roseltorg.ru Roseltorg procurement productive low 1 catalog-ru-20260623 Electronic trading platform 1
|
||||
rts-tender.ru RTS Tender procurement productive low 1 catalog-ru-20260623 Electronic trading platform 1
|
||||
fabrikant.ru Fabrikant procurement productive low 1 catalog-ru-20260623 Electronic trading platform 1
|
||||
spark-interfax.ru SPARK Interfax business_reference productive low 1 catalog-ru-20260623 Counterparty checks 1
|
||||
rusprofile.ru Rusprofile business_reference productive low 1 catalog-ru-20260623 Counterparty checks 1
|
||||
checko.ru Checko business_reference productive low 1 catalog-ru-20260623 Counterparty checks 1
|
||||
zachestnyibiznes.ru Za Chestny Biznes business_reference productive low 1 catalog-ru-20260623 Counterparty checks 1
|
||||
list-org.com List-Org business_reference neutral medium 1 catalog-ru-20260623 Counterparty checks with mixed data quality 1
|
||||
garant.ru Garant legal_reference productive low 1 catalog-ru-20260623 Legal reference system 1
|
||||
consultant.ru ConsultantPlus legal_reference productive low 1 catalog-ru-20260623 Legal reference system 1
|
||||
mail.yandex.ru Yandex Mail mail productive low 1 catalog-ru-20260623 Business mail when used with org account 1
|
||||
360.yandex.ru Yandex 360 mail productive low 1 catalog-ru-20260623 Business mail and documents 1
|
||||
mail.ru Mail.ru mail neutral medium 1 catalog-ru-20260623 Mail portal; classify account policy separately if needed 1
|
||||
e.mail.ru Mail.ru webmail mail neutral medium 1 catalog-ru-20260623 Mail.ru webmail 1
|
||||
biz.mail.ru Mail.ru business mail productive low 1 catalog-ru-20260623 Business mail 1
|
||||
corp.mail.ru Mail.ru corporate mail productive low 1 catalog-ru-20260623 Corporate services 1
|
||||
gmail.com Gmail mail neutral medium 1 catalog-ru-20260623 External mail 1
|
||||
outlook.office.com Outlook Web mail productive low 1 catalog-ru-20260623 Business mail 1
|
||||
yandex.ru Yandex search neutral low 1 catalog-ru-20260623 Search portal 1
|
||||
ya.ru Yandex search neutral low 1 catalog-ru-20260623 Search portal 1
|
||||
google.com Google Search search neutral low 1 catalog-ru-20260623 Search portal 1
|
||||
2gis.ru 2GIS maps_reference productive low 1 catalog-ru-20260623 Maps and organization reference 1
|
||||
maps.yandex.ru Yandex Maps maps_reference productive low 1 catalog-ru-20260623 Maps and organization reference 1
|
||||
dadata.ru DaData business_reference productive low 1 catalog-ru-20260623 Address and company reference 1
|
||||
disk.yandex.ru Yandex Disk cloud_docs productive medium 1 catalog-ru-20260623 Cloud documents; data handling policy applies 1
|
||||
docs.yandex.ru Yandex Documents cloud_docs productive medium 1 catalog-ru-20260623 Cloud documents; data handling policy applies 1
|
||||
cloud.mail.ru Cloud Mail.ru cloud_docs productive medium 1 catalog-ru-20260623 Cloud storage; data handling policy applies 1
|
||||
docs.google.com Google Docs cloud_docs productive medium 1 catalog-ru-20260623 Cloud documents; data handling policy applies 1
|
||||
drive.google.com Google Drive cloud_docs productive medium 1 catalog-ru-20260623 Cloud storage; data handling policy applies 1
|
||||
office.com Microsoft 365 cloud_docs productive medium 1 catalog-ru-20260623 Cloud documents; data handling policy applies 1
|
||||
microsoft365.com Microsoft 365 cloud_docs productive medium 1 catalog-ru-20260623 Cloud documents; data handling policy applies 1
|
||||
web.telegram.org Telegram Web communication neutral medium 1 catalog-ru-20260623 Messenger; business use depends on policy 1
|
||||
t.me Telegram links communication neutral medium 1 catalog-ru-20260623 Messenger links 1
|
||||
telegram.org Telegram communication neutral medium 1 catalog-ru-20260623 Messenger 1
|
||||
web.whatsapp.com WhatsApp Web communication neutral medium 1 catalog-ru-20260623 Messenger; business use depends on policy 1
|
||||
whatsapp.com WhatsApp communication neutral medium 1 catalog-ru-20260623 Messenger 1
|
||||
teams.microsoft.com Microsoft Teams communication productive low 1 catalog-ru-20260623 Business communication 1
|
||||
meet.google.com Google Meet communication productive low 1 catalog-ru-20260623 Business communication 1
|
||||
zoom.us Zoom communication productive low 1 catalog-ru-20260623 Business communication 1
|
||||
github.com GitHub developer productive low 1 catalog-ru-20260623 Development and documentation workflow 1
|
||||
gitlab.com GitLab developer productive low 1 catalog-ru-20260623 Development workflow 1
|
||||
stackoverflow.com Stack Overflow developer productive low 1 catalog-ru-20260623 Technical reference 1
|
||||
learn.microsoft.com Microsoft Learn developer productive low 1 catalog-ru-20260623 Technical documentation 1
|
||||
docs.microsoft.com Microsoft Docs developer productive low 1 catalog-ru-20260623 Technical documentation legacy host 1
|
||||
microsoft.com Microsoft developer neutral low 1 catalog-ru-20260623 Vendor documentation and downloads 1
|
||||
docker.com Docker developer productive low 1 catalog-ru-20260623 Developer tooling documentation 1
|
||||
software.qnap.com QNAP Software work_service neutral low 1 catalog-ru-20260623 Vendor software/download portal for admin work 1
|
||||
cryptopro.ru CryptoPro security_crypto productive low 1 catalog-ru-20260623 CryptoPro documentation and downloads 1
|
||||
www.cryptopro.ru CryptoPro security_crypto productive low 1 catalog-ru-20260623 CryptoPro documentation and downloads 1
|
||||
chatgpt.com ChatGPT ai_assistant productive medium 1 catalog-ru-20260623 AI assistant; data handling policy applies 1
|
||||
chat.openai.com ChatGPT legacy ai_assistant productive medium 1 catalog-ru-20260623 AI assistant; data handling policy applies 1
|
||||
giga.chat GigaChat ai_assistant productive medium 1 catalog-ru-20260623 AI assistant; data handling policy applies 1
|
||||
alice.yandex.ru Yandex Alice ai_assistant productive medium 1 catalog-ru-20260623 AI assistant; data handling policy applies 1
|
||||
wildberries.ru Wildberries marketplace neutral medium 1 catalog-ru-20260623 Marketplace; business purchases possible 1
|
||||
ozon.ru Ozon marketplace neutral medium 1 catalog-ru-20260623 Marketplace; business purchases possible 1
|
||||
market.yandex.ru Yandex Market marketplace neutral medium 1 catalog-ru-20260623 Marketplace; business purchases possible 1
|
||||
avito.ru Avito marketplace neutral medium 1 catalog-ru-20260623 Marketplace; business use depends on policy 1
|
||||
dns-shop.ru DNS Shop marketplace neutral medium 1 catalog-ru-20260623 IT and office equipment purchases 1
|
||||
citilink.ru Citilink marketplace neutral medium 1 catalog-ru-20260623 IT and office equipment purchases 1
|
||||
vk.com VK social non_productive medium 0 catalog-ru-20260623 Social network; whitelist business communities separately if needed 1
|
||||
ok.ru Odnoklassniki social non_productive medium 0 catalog-ru-20260623 Social network 1
|
||||
dzen.ru Dzen news non_productive medium 0 catalog-ru-20260623 News/feed portal 1
|
||||
rbc.ru RBC news neutral medium 1 catalog-ru-20260623 Business news; review by department policy 1
|
||||
ria.ru RIA Novosti news neutral medium 1 catalog-ru-20260623 News; review by department policy 1
|
||||
lenta.ru Lenta news non_productive medium 0 catalog-ru-20260623 News/feed portal 1
|
||||
youtube.com YouTube media non_productive medium 0 catalog-ru-20260623 Video platform; business use requires URL-level exception 1
|
||||
rutube.ru Rutube media non_productive medium 0 catalog-ru-20260623 Video platform 1
|
||||
vkvideo.ru VK Video media non_productive medium 0 catalog-ru-20260623 Video platform 1
|
||||
music.yandex.ru Yandex Music media non_productive medium 0 catalog-ru-20260623 Music streaming 1
|
||||
kinopoisk.ru Kinopoisk media non_productive medium 0 catalog-ru-20260623 Entertainment streaming 1
|
||||
twitch.tv Twitch media non_productive high 0 catalog-ru-20260623 Entertainment streaming 1
|
||||
example.com Example Domain test_reference neutral low 0 catalog-ru-20260623 Test/reference domain 1
|
||||
unknown.example Unknown example test_reference neutral low 0 catalog-ru-20260623 Test/reference domain from demo data 1
|
||||
|
@@ -0,0 +1,8 @@
|
||||
host_name user_login user_domain employee_id employee_name department branch position source is_active
|
||||
SHARKON2025 user1 sharkon2025 sharkon2025\\user1 user1 tsj tsj RDP user catalog-p5 1
|
||||
SHARKON2025 user4 sharkon2025 sharkon2025\\user4 user4 tsj tsj RDP user catalog-p5 1
|
||||
SHARKON2025 user5 sharkon2025 sharkon2025\\user5 user5 tsj tsj RDP user catalog-p5 1
|
||||
SHARKON2025 Администратор sharkon2025 sharkon2025\\Администратор Администратор it_admin tsj Windows administrator catalog-p5 1
|
||||
SHARKON2025 администратор sharkon2025 sharkon2025\\Администратор Администратор it_admin tsj Windows administrator legacy lowercase alias catalog-p5 1
|
||||
ws-001 ivanov corp E001 Иванов И.И. Бухгалтерия Филиал 1 Бухгалтер demo 1
|
||||
ws-002 petrova corp E002 Петрова П.П. Операционный отдел Филиал 1 Оператор demo 1
|
||||
|
@@ -0,0 +1 @@
|
||||
CREATE DATABASE IF NOT EXISTS aw_workforce;
|
||||
@@ -0,0 +1,32 @@
|
||||
CREATE TABLE IF NOT EXISTS aw_workforce.aw_window_events
|
||||
(
|
||||
event_time DateTime,
|
||||
host_name String,
|
||||
user_login String,
|
||||
process_name String,
|
||||
window_title String,
|
||||
duration_sec UInt32,
|
||||
source_bucket LowCardinality(String),
|
||||
source_event_id String,
|
||||
ingested_at DateTime DEFAULT now()
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
PARTITION BY toYYYYMM(event_time)
|
||||
ORDER BY (event_time, host_name, user_login, process_name, source_event_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS aw_workforce.aw_browser_events
|
||||
(
|
||||
event_time DateTime,
|
||||
host_name String,
|
||||
user_login String,
|
||||
browser_name String,
|
||||
url String,
|
||||
title String,
|
||||
duration_sec UInt32,
|
||||
source_bucket LowCardinality(String),
|
||||
source_event_id String,
|
||||
ingested_at DateTime DEFAULT now()
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
PARTITION BY toYYYYMM(event_time)
|
||||
ORDER BY (event_time, host_name, user_login, browser_name, source_event_id);
|
||||
@@ -0,0 +1,145 @@
|
||||
CREATE TABLE IF NOT EXISTS aw_workforce.dim_workstation_user
|
||||
(
|
||||
host_name String,
|
||||
user_login String,
|
||||
user_domain String,
|
||||
|
||||
employee_id String,
|
||||
employee_name String,
|
||||
department String,
|
||||
branch String,
|
||||
position String,
|
||||
|
||||
source LowCardinality(String),
|
||||
is_active UInt8 DEFAULT 1,
|
||||
updated_at DateTime DEFAULT now()
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(updated_at)
|
||||
ORDER BY (host_name, user_login);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS aw_workforce.dim_application_category
|
||||
(
|
||||
process_name String,
|
||||
|
||||
application_name String,
|
||||
vendor String,
|
||||
category LowCardinality(String),
|
||||
productivity_class LowCardinality(String),
|
||||
risk_level LowCardinality(String),
|
||||
|
||||
is_system UInt8 DEFAULT 0,
|
||||
is_active UInt8 DEFAULT 1,
|
||||
source LowCardinality(String),
|
||||
comment String,
|
||||
updated_at DateTime DEFAULT now()
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(updated_at)
|
||||
ORDER BY process_name;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS aw_workforce.dim_domain_category
|
||||
(
|
||||
domain String,
|
||||
|
||||
site_name String,
|
||||
category LowCardinality(String),
|
||||
productivity_class LowCardinality(String),
|
||||
risk_level LowCardinality(String),
|
||||
business_allowed UInt8 DEFAULT 0,
|
||||
|
||||
source LowCardinality(String),
|
||||
comment String,
|
||||
is_active UInt8 DEFAULT 1,
|
||||
updated_at DateTime DEFAULT now()
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(updated_at)
|
||||
ORDER BY domain;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS aw_workforce.dim_url_rule
|
||||
(
|
||||
rule_id String,
|
||||
domain String,
|
||||
path_pattern String,
|
||||
|
||||
category LowCardinality(String),
|
||||
productivity_class LowCardinality(String),
|
||||
risk_level LowCardinality(String),
|
||||
|
||||
priority UInt16 DEFAULT 100,
|
||||
is_active UInt8 DEFAULT 1,
|
||||
comment String,
|
||||
updated_at DateTime DEFAULT now()
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(updated_at)
|
||||
ORDER BY (domain, priority, rule_id);
|
||||
|
||||
DROP DICTIONARY IF EXISTS aw_workforce.dict_workstation_user;
|
||||
|
||||
CREATE DICTIONARY aw_workforce.dict_workstation_user
|
||||
(
|
||||
host_name String,
|
||||
user_login String,
|
||||
user_domain String,
|
||||
|
||||
employee_id String,
|
||||
employee_name String,
|
||||
department String,
|
||||
branch String,
|
||||
position String,
|
||||
is_active UInt8
|
||||
)
|
||||
PRIMARY KEY host_name, user_login
|
||||
SOURCE(CLICKHOUSE(
|
||||
USER 'aw_workforce_dict'
|
||||
PASSWORD ''
|
||||
DB 'aw_workforce'
|
||||
TABLE 'dim_workstation_user'
|
||||
))
|
||||
LAYOUT(COMPLEX_KEY_HASHED())
|
||||
LIFETIME(MIN 3600 MAX 86400);
|
||||
|
||||
DROP DICTIONARY IF EXISTS aw_workforce.dict_application_category;
|
||||
|
||||
CREATE DICTIONARY aw_workforce.dict_application_category
|
||||
(
|
||||
process_name String,
|
||||
|
||||
application_name String,
|
||||
vendor String,
|
||||
category String,
|
||||
productivity_class String,
|
||||
risk_level String,
|
||||
is_system UInt8,
|
||||
is_active UInt8
|
||||
)
|
||||
PRIMARY KEY process_name
|
||||
SOURCE(CLICKHOUSE(
|
||||
USER 'aw_workforce_dict'
|
||||
PASSWORD ''
|
||||
DB 'aw_workforce'
|
||||
TABLE 'dim_application_category'
|
||||
))
|
||||
LAYOUT(HASHED())
|
||||
LIFETIME(MIN 3600 MAX 86400);
|
||||
|
||||
DROP DICTIONARY IF EXISTS aw_workforce.dict_domain_category;
|
||||
|
||||
CREATE DICTIONARY aw_workforce.dict_domain_category
|
||||
(
|
||||
domain String,
|
||||
|
||||
site_name String,
|
||||
category String,
|
||||
productivity_class String,
|
||||
risk_level String,
|
||||
business_allowed UInt8,
|
||||
is_active UInt8
|
||||
)
|
||||
PRIMARY KEY domain
|
||||
SOURCE(CLICKHOUSE(
|
||||
USER 'aw_workforce_dict'
|
||||
PASSWORD ''
|
||||
DB 'aw_workforce'
|
||||
TABLE 'dim_domain_category'
|
||||
))
|
||||
LAYOUT(HASHED())
|
||||
LIFETIME(MIN 3600 MAX 86400);
|
||||
@@ -0,0 +1,158 @@
|
||||
CREATE TABLE IF NOT EXISTS aw_workforce.agg_workforce_productivity_hourly
|
||||
(
|
||||
bucket_start DateTime,
|
||||
event_date Date,
|
||||
|
||||
branch LowCardinality(String),
|
||||
department LowCardinality(String),
|
||||
|
||||
activity_type LowCardinality(String),
|
||||
category LowCardinality(String),
|
||||
productivity_class LowCardinality(String),
|
||||
|
||||
duration_sec UInt64,
|
||||
event_count UInt64,
|
||||
unknown_subject_events UInt64,
|
||||
unknown_category_events UInt64
|
||||
)
|
||||
ENGINE = SummingMergeTree((
|
||||
duration_sec,
|
||||
event_count,
|
||||
unknown_subject_events,
|
||||
unknown_category_events
|
||||
))
|
||||
PARTITION BY toYYYYMM(event_date)
|
||||
ORDER BY (
|
||||
event_date,
|
||||
bucket_start,
|
||||
branch,
|
||||
department,
|
||||
activity_type,
|
||||
productivity_class,
|
||||
category
|
||||
);
|
||||
|
||||
DROP VIEW IF EXISTS aw_workforce.mv_desktop_productivity_hourly;
|
||||
|
||||
CREATE MATERIALIZED VIEW aw_workforce.mv_desktop_productivity_hourly
|
||||
TO aw_workforce.agg_workforce_productivity_hourly
|
||||
AS
|
||||
SELECT
|
||||
toStartOfHour(event_time) AS bucket_start,
|
||||
toDate(event_time) AS event_date,
|
||||
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS branch,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS department,
|
||||
|
||||
'desktop' AS activity_type,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_application_category', 'is_active', process_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_application_category', 'category', process_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_application_category', 'category', process_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS category,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_application_category', 'is_active', process_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_application_category', 'productivity_class', process_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_application_category', 'productivity_class', process_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS productivity_class,
|
||||
|
||||
toUInt64(sum(duration_sec)) AS duration_sec,
|
||||
toUInt64(count()) AS event_count,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'employee_name', (host_name, user_login), '') != '',
|
||||
0,
|
||||
1
|
||||
))) AS unknown_subject_events,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_application_category', 'is_active', process_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_application_category', 'category', process_name, '') NOT IN ('', 'unknown'),
|
||||
0,
|
||||
1
|
||||
))) AS unknown_category_events
|
||||
FROM aw_workforce.aw_window_events
|
||||
GROUP BY
|
||||
bucket_start,
|
||||
event_date,
|
||||
branch,
|
||||
department,
|
||||
activity_type,
|
||||
category,
|
||||
productivity_class;
|
||||
|
||||
DROP VIEW IF EXISTS aw_workforce.mv_browser_productivity_hourly;
|
||||
|
||||
CREATE MATERIALIZED VIEW aw_workforce.mv_browser_productivity_hourly
|
||||
TO aw_workforce.agg_workforce_productivity_hourly
|
||||
AS
|
||||
WITH
|
||||
lowerUTF8(
|
||||
domain(if(position(url, '://') = 0, concat('http://', url), url))
|
||||
) AS domain_name
|
||||
SELECT
|
||||
toStartOfHour(event_time) AS bucket_start,
|
||||
toDate(event_time) AS event_date,
|
||||
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'branch', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS branch,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'department', (host_name, user_login), 'unknown'),
|
||||
'unknown'
|
||||
) AS department,
|
||||
|
||||
'browser' AS activity_type,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_domain_category', 'is_active', domain_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_domain_category', 'category', domain_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_domain_category', 'category', domain_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS category,
|
||||
if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_domain_category', 'is_active', domain_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_domain_category', 'productivity_class', domain_name, '') != '',
|
||||
dictGetStringOrDefault('aw_workforce.dict_domain_category', 'productivity_class', domain_name, 'unknown'),
|
||||
'unknown'
|
||||
) AS productivity_class,
|
||||
|
||||
toUInt64(sum(duration_sec)) AS duration_sec,
|
||||
toUInt64(count()) AS event_count,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_workstation_user', 'is_active', (host_name, user_login), 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_workstation_user', 'employee_name', (host_name, user_login), '') != '',
|
||||
0,
|
||||
1
|
||||
))) AS unknown_subject_events,
|
||||
toUInt64(sum(if(
|
||||
dictGetUInt8OrDefault('aw_workforce.dict_domain_category', 'is_active', domain_name, 0) = 1
|
||||
AND dictGetStringOrDefault('aw_workforce.dict_domain_category', 'category', domain_name, '') NOT IN ('', 'unknown'),
|
||||
0,
|
||||
1
|
||||
))) AS unknown_category_events
|
||||
FROM aw_workforce.aw_browser_events
|
||||
WHERE domain_name != ''
|
||||
GROUP BY
|
||||
bucket_start,
|
||||
event_date,
|
||||
branch,
|
||||
department,
|
||||
activity_type,
|
||||
category,
|
||||
productivity_class;
|
||||
@@ -0,0 +1,105 @@
|
||||
CREATE OR REPLACE VIEW aw_workforce.v_workforce_productivity_daily AS
|
||||
SELECT
|
||||
event_date,
|
||||
branch,
|
||||
department,
|
||||
activity_type,
|
||||
category,
|
||||
productivity_class,
|
||||
sum(duration_sec) AS duration_sec,
|
||||
sum(event_count) AS event_count,
|
||||
sum(unknown_subject_events) AS unknown_subject_events,
|
||||
sum(unknown_category_events) AS unknown_category_events
|
||||
FROM aw_workforce.agg_workforce_productivity_hourly
|
||||
GROUP BY
|
||||
event_date,
|
||||
branch,
|
||||
department,
|
||||
activity_type,
|
||||
category,
|
||||
productivity_class;
|
||||
|
||||
CREATE OR REPLACE VIEW aw_workforce.v_workforce_unknown_subjects AS
|
||||
SELECT
|
||||
host_name,
|
||||
user_login,
|
||||
count() AS events,
|
||||
sum(duration_sec) AS duration_sec
|
||||
FROM aw_workforce.aw_window_events
|
||||
WHERE dictGetUInt8OrDefault(
|
||||
'aw_workforce.dict_workstation_user',
|
||||
'is_active',
|
||||
(host_name, user_login),
|
||||
0
|
||||
) != 1
|
||||
OR dictGetStringOrDefault(
|
||||
'aw_workforce.dict_workstation_user',
|
||||
'employee_name',
|
||||
(host_name, user_login),
|
||||
''
|
||||
) = ''
|
||||
GROUP BY
|
||||
host_name,
|
||||
user_login
|
||||
ORDER BY duration_sec DESC;
|
||||
|
||||
CREATE OR REPLACE VIEW aw_workforce.v_workforce_unknown_processes AS
|
||||
SELECT
|
||||
process_name,
|
||||
count() AS events,
|
||||
sum(duration_sec) AS duration_sec
|
||||
FROM aw_workforce.aw_window_events
|
||||
WHERE dictGetUInt8OrDefault(
|
||||
'aw_workforce.dict_application_category',
|
||||
'is_active',
|
||||
process_name,
|
||||
0
|
||||
) != 1
|
||||
OR dictGetStringOrDefault(
|
||||
'aw_workforce.dict_application_category',
|
||||
'category',
|
||||
process_name,
|
||||
''
|
||||
) IN ('', 'unknown')
|
||||
GROUP BY process_name
|
||||
ORDER BY duration_sec DESC;
|
||||
|
||||
CREATE OR REPLACE VIEW aw_workforce.v_workforce_unknown_domains AS
|
||||
WITH
|
||||
lowerUTF8(
|
||||
domain(if(position(url, '://') = 0, concat('http://', url), url))
|
||||
) AS domain_name
|
||||
SELECT
|
||||
domain_name,
|
||||
count() AS events,
|
||||
sum(duration_sec) AS duration_sec
|
||||
FROM aw_workforce.aw_browser_events
|
||||
WHERE domain_name != ''
|
||||
AND (
|
||||
dictGetUInt8OrDefault(
|
||||
'aw_workforce.dict_domain_category',
|
||||
'is_active',
|
||||
domain_name,
|
||||
0
|
||||
) != 1
|
||||
OR dictGetStringOrDefault(
|
||||
'aw_workforce.dict_domain_category',
|
||||
'category',
|
||||
domain_name,
|
||||
''
|
||||
) IN ('', 'unknown')
|
||||
)
|
||||
GROUP BY domain_name
|
||||
ORDER BY duration_sec DESC;
|
||||
|
||||
CREATE OR REPLACE VIEW aw_workforce.v_workforce_unknown_quality_daily AS
|
||||
SELECT
|
||||
event_date,
|
||||
sum(event_count) AS events,
|
||||
sum(unknown_subject_events) AS unknown_subject_events,
|
||||
round(unknown_subject_events / nullIf(events, 0), 4) AS unknown_subject_ratio,
|
||||
sum(unknown_category_events) AS unknown_category_events,
|
||||
round(unknown_category_events / nullIf(events, 0), 4) AS unknown_category_ratio
|
||||
FROM aw_workforce.agg_workforce_productivity_hourly
|
||||
GROUP BY event_date
|
||||
ORDER BY event_date DESC;
|
||||
@@ -0,0 +1,18 @@
|
||||
<clickhouse>
|
||||
<users>
|
||||
<default>
|
||||
<networks replace="replace">
|
||||
<ip>::/0</ip>
|
||||
</networks>
|
||||
</default>
|
||||
<aw_workforce_dict>
|
||||
<no_password/>
|
||||
<networks>
|
||||
<ip>127.0.0.1</ip>
|
||||
<ip>::1</ip>
|
||||
</networks>
|
||||
<profile>readonly</profile>
|
||||
<quota>default</quota>
|
||||
</aw_workforce_dict>
|
||||
</users>
|
||||
</clickhouse>
|
||||
@@ -0,0 +1,17 @@
|
||||
services:
|
||||
clickhouse:
|
||||
image: clickhouse/clickhouse-server:24.8
|
||||
container_name: aw-rus-workforce-clickhouse
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
CLICKHOUSE_DB: ${CLICKHOUSE_DB:-aw_workforce}
|
||||
ports:
|
||||
- "${CLICKHOUSE_HTTP_BIND:-127.0.0.1}:${CLICKHOUSE_PORT:-8124}:8123"
|
||||
- "${CLICKHOUSE_NATIVE_BIND:-127.0.0.1}:${CLICKHOUSE_NATIVE_PORT:-9001}:9000"
|
||||
volumes:
|
||||
- clickhouse_workforce_data:/var/lib/clickhouse
|
||||
- ./clickhouse/init:/docker-entrypoint-initdb.d:ro
|
||||
- ./clickhouse/users.d/99-aw-workforce-local.xml:/etc/clickhouse-server/users.d/99-aw-workforce-local.xml:ro
|
||||
|
||||
volumes:
|
||||
clickhouse_workforce_data:
|
||||
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
CATALOG_DIR="${CATALOG_DIR:-$ROOT_DIR/catalog}"
|
||||
|
||||
CLICKHOUSE_CONTAINER="${CLICKHOUSE_CONTAINER:-aw-rus-workforce-clickhouse}"
|
||||
CLICKHOUSE_DATABASE="${CLICKHOUSE_DATABASE:-aw_workforce}"
|
||||
CLICKHOUSE_USER="${CLICKHOUSE_USER:-}"
|
||||
CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-}"
|
||||
CLICKHOUSE_CLIENT_BIN="${CLICKHOUSE_CLIENT_BIN:-clickhouse-client}"
|
||||
CLICKHOUSE_READY_TIMEOUT_SEC="${CLICKHOUSE_READY_TIMEOUT_SEC:-60}"
|
||||
REBUILD_AGGREGATES="${REBUILD_AGGREGATES:-0}"
|
||||
|
||||
client_auth_args=()
|
||||
if [[ -n "$CLICKHOUSE_USER" ]]; then
|
||||
client_auth_args+=(--user "$CLICKHOUSE_USER")
|
||||
fi
|
||||
if [[ -n "$CLICKHOUSE_PASSWORD" ]]; then
|
||||
client_auth_args+=(--password "$CLICKHOUSE_PASSWORD")
|
||||
fi
|
||||
|
||||
run_query() {
|
||||
local query="$1"
|
||||
|
||||
if docker ps --format '{{.Names}}' | grep -Fxq "$CLICKHOUSE_CONTAINER"; then
|
||||
docker exec -i "$CLICKHOUSE_CONTAINER" clickhouse-client "${client_auth_args[@]}" \
|
||||
--database "$CLICKHOUSE_DATABASE" --query "$query"
|
||||
return
|
||||
fi
|
||||
|
||||
if command -v "$CLICKHOUSE_CLIENT_BIN" >/dev/null 2>&1; then
|
||||
"$CLICKHOUSE_CLIENT_BIN" "${client_auth_args[@]}" \
|
||||
--database "$CLICKHOUSE_DATABASE" --query "$query"
|
||||
return
|
||||
fi
|
||||
|
||||
printf 'No running ClickHouse container "%s" and no %s in PATH\n' \
|
||||
"$CLICKHOUSE_CONTAINER" "$CLICKHOUSE_CLIENT_BIN" >&2
|
||||
return 127
|
||||
}
|
||||
|
||||
run_query_file() {
|
||||
local query_file="$1"
|
||||
|
||||
if docker ps --format '{{.Names}}' | grep -Fxq "$CLICKHOUSE_CONTAINER"; then
|
||||
docker exec -i "$CLICKHOUSE_CONTAINER" clickhouse-client "${client_auth_args[@]}" \
|
||||
--multiquery <"$query_file"
|
||||
return
|
||||
fi
|
||||
|
||||
if command -v "$CLICKHOUSE_CLIENT_BIN" >/dev/null 2>&1; then
|
||||
"$CLICKHOUSE_CLIENT_BIN" "${client_auth_args[@]}" --multiquery <"$query_file"
|
||||
return
|
||||
fi
|
||||
|
||||
printf 'No running ClickHouse container "%s" and no %s in PATH\n' \
|
||||
"$CLICKHOUSE_CONTAINER" "$CLICKHOUSE_CLIENT_BIN" >&2
|
||||
return 127
|
||||
}
|
||||
|
||||
run_insert_file() {
|
||||
local query="$1"
|
||||
local data_file="$2"
|
||||
|
||||
if docker ps --format '{{.Names}}' | grep -Fxq "$CLICKHOUSE_CONTAINER"; then
|
||||
docker exec -i "$CLICKHOUSE_CONTAINER" clickhouse-client "${client_auth_args[@]}" \
|
||||
--database "$CLICKHOUSE_DATABASE" --query "$query" <"$data_file"
|
||||
return
|
||||
fi
|
||||
|
||||
if command -v "$CLICKHOUSE_CLIENT_BIN" >/dev/null 2>&1; then
|
||||
"$CLICKHOUSE_CLIENT_BIN" "${client_auth_args[@]}" \
|
||||
--database "$CLICKHOUSE_DATABASE" --query "$query" <"$data_file"
|
||||
return
|
||||
fi
|
||||
|
||||
printf 'No running ClickHouse container "%s" and no %s in PATH\n' \
|
||||
"$CLICKHOUSE_CONTAINER" "$CLICKHOUSE_CLIENT_BIN" >&2
|
||||
return 127
|
||||
}
|
||||
|
||||
wait_for_clickhouse() {
|
||||
local deadline
|
||||
deadline=$((SECONDS + CLICKHOUSE_READY_TIMEOUT_SEC))
|
||||
while (( SECONDS < deadline )); do
|
||||
if run_query "SELECT 1" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
printf 'ClickHouse is not ready after %s seconds\n' "$CLICKHOUSE_READY_TIMEOUT_SEC" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
require_file() {
|
||||
local path="$1"
|
||||
if [[ ! -f "$path" ]]; then
|
||||
printf 'Required catalog file is missing: %s\n' "$path" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_clickhouse
|
||||
|
||||
require_file "$CATALOG_DIR/workstation_users.tsv"
|
||||
require_file "$CATALOG_DIR/application_categories.tsv"
|
||||
require_file "$CATALOG_DIR/domain_categories.tsv"
|
||||
|
||||
printf '[catalog] truncate dimension tables\n'
|
||||
run_query "TRUNCATE TABLE $CLICKHOUSE_DATABASE.dim_workstation_user"
|
||||
run_query "TRUNCATE TABLE $CLICKHOUSE_DATABASE.dim_application_category"
|
||||
run_query "TRUNCATE TABLE $CLICKHOUSE_DATABASE.dim_domain_category"
|
||||
|
||||
printf '[catalog] load workstation users\n'
|
||||
run_insert_file "
|
||||
INSERT INTO $CLICKHOUSE_DATABASE.dim_workstation_user
|
||||
(host_name, user_login, user_domain, employee_id, employee_name, department, branch, position, source, is_active)
|
||||
FORMAT TabSeparatedWithNames
|
||||
" "$CATALOG_DIR/workstation_users.tsv"
|
||||
|
||||
printf '[catalog] load application categories\n'
|
||||
run_insert_file "
|
||||
INSERT INTO $CLICKHOUSE_DATABASE.dim_application_category
|
||||
(process_name, application_name, vendor, category, productivity_class, risk_level, is_system, source, comment, is_active)
|
||||
FORMAT TabSeparatedWithNames
|
||||
" "$CATALOG_DIR/application_categories.tsv"
|
||||
|
||||
printf '[catalog] load domain categories\n'
|
||||
run_insert_file "
|
||||
INSERT INTO $CLICKHOUSE_DATABASE.dim_domain_category
|
||||
(domain, site_name, category, productivity_class, risk_level, business_allowed, source, comment, is_active)
|
||||
FORMAT TabSeparatedWithNames
|
||||
" "$CATALOG_DIR/domain_categories.tsv"
|
||||
|
||||
printf '[catalog] reload dictionaries\n'
|
||||
run_query "SYSTEM RELOAD DICTIONARY $CLICKHOUSE_DATABASE.dict_workstation_user"
|
||||
run_query "SYSTEM RELOAD DICTIONARY $CLICKHOUSE_DATABASE.dict_application_category"
|
||||
run_query "SYSTEM RELOAD DICTIONARY $CLICKHOUSE_DATABASE.dict_domain_category"
|
||||
|
||||
if [[ "$REBUILD_AGGREGATES" == "1" ]]; then
|
||||
printf '[catalog] rebuild aggregates\n'
|
||||
run_query_file "$ROOT_DIR/admin/rebuild_aggregates.sql"
|
||||
fi
|
||||
|
||||
printf '[catalog] dictionary status\n'
|
||||
run_query "
|
||||
SELECT name, status, last_exception
|
||||
FROM system.dictionaries
|
||||
WHERE database = '$CLICKHOUSE_DATABASE'
|
||||
AND name IN ('dict_workstation_user', 'dict_application_category', 'dict_domain_category')
|
||||
ORDER BY name
|
||||
FORMAT PrettyCompact
|
||||
"
|
||||
|
||||
printf '[catalog] raw unknown summary\n'
|
||||
run_query "
|
||||
SELECT 'subjects' AS area, count() AS rows
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_unknown_subjects
|
||||
UNION ALL
|
||||
SELECT 'processes' AS area, count() AS rows
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_unknown_processes
|
||||
UNION ALL
|
||||
SELECT 'domains' AS area, count() AS rows
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_unknown_domains
|
||||
FORMAT PrettyCompact
|
||||
"
|
||||
@@ -0,0 +1,14 @@
|
||||
AW_WORKFORCE_AW_URL=http://10.10.10.13:5600/api/0
|
||||
AW_WORKFORCE_CLICKHOUSE_URL=http://10.10.10.2:8123
|
||||
AW_WORKFORCE_CLICKHOUSE_DATABASE=aw_workforce
|
||||
AW_WORKFORCE_CLICKHOUSE_USER=
|
||||
AW_WORKFORCE_CLICKHOUSE_PASSWORD=
|
||||
AW_WORKFORCE_HOST=SHARKON2025
|
||||
AW_WORKFORCE_STATE_PATH=/var/lib/aw-workforce-ingest/state.json
|
||||
AW_WORKFORCE_OVERLAP_SECONDS=300
|
||||
AW_WORKFORCE_RETRY_ATTEMPTS=3
|
||||
AW_WORKFORCE_RETRY_BACKOFF_MS=1000
|
||||
AW_WORKFORCE_FAIL_ON_EMPTY=false
|
||||
AW_WORKFORCE_JSON=true
|
||||
no_proxy=localhost,127.0.0.1,10.10.10.13,10.10.10.2,10.10.10.0/24
|
||||
NO_PROXY=localhost,127.0.0.1,10.10.10.13,10.10.10.2,10.10.10.0/24
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=AW-rus workforce ClickHouse ingest
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/opt/activitywatch/clickhouse-workforce
|
||||
EnvironmentFile=-/etc/activitywatch/aw-workforce-ingest.env
|
||||
ExecStart=/usr/local/bin/aw-workforce-ingest
|
||||
TimeoutStartSec=20min
|
||||
User=root
|
||||
Group=root
|
||||
Nice=5
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=aw-workforce-ingest
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Run AW-rus workforce ClickHouse ingest every 5 minutes
|
||||
|
||||
[Timer]
|
||||
OnBootSec=2min
|
||||
OnUnitActiveSec=5min
|
||||
AccuracySec=30s
|
||||
RandomizedDelaySec=30s
|
||||
Persistent=true
|
||||
Unit=aw-workforce-ingest.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="${AW_WORKFORCE_ROOT:-/opt/activitywatch/clickhouse-workforce}"
|
||||
ENV_DIR="${AW_WORKFORCE_ENV_DIR:-/etc/activitywatch}"
|
||||
STATE_DIR="${AW_WORKFORCE_STATE_DIR:-/var/lib/aw-workforce-ingest}"
|
||||
SYSTEMD_DIR="${AW_WORKFORCE_SYSTEMD_DIR:-/etc/systemd/system}"
|
||||
|
||||
install -d -m 0755 "$ROOT" "$ENV_DIR" "$STATE_DIR" "$SYSTEMD_DIR"
|
||||
|
||||
if [[ ! -f "$ENV_DIR/aw-workforce-ingest.env" ]]; then
|
||||
install -m 0640 "$SCRIPT_DIR/aw-workforce-ingest.env.example" \
|
||||
"$ENV_DIR/aw-workforce-ingest.env"
|
||||
fi
|
||||
|
||||
install -m 0644 "$SCRIPT_DIR/aw-workforce-ingest.service" \
|
||||
"$SYSTEMD_DIR/aw-workforce-ingest.service"
|
||||
install -m 0644 "$SCRIPT_DIR/aw-workforce-ingest.timer" \
|
||||
"$SYSTEMD_DIR/aw-workforce-ingest.timer"
|
||||
|
||||
if command -v systemctl >/dev/null 2>&1; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
Installed aw-workforce-ingest runtime files.
|
||||
|
||||
Next manual deployment steps:
|
||||
install -m 0755 <built aw-workforce-ingest binary> /usr/local/bin/aw-workforce-ingest
|
||||
edit $ENV_DIR/aw-workforce-ingest.env
|
||||
systemctl enable --now aw-workforce-ingest.timer
|
||||
systemctl start aw-workforce-ingest.service
|
||||
EOF
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
|
||||
CLICKHOUSE_CONTAINER="${CLICKHOUSE_CONTAINER:-aw-rus-workforce-clickhouse}"
|
||||
CLICKHOUSE_DATABASE="${CLICKHOUSE_DATABASE:-aw_workforce}"
|
||||
CLICKHOUSE_CLIENT_BIN="${CLICKHOUSE_CLIENT_BIN:-clickhouse-client}"
|
||||
LIMIT="${LIMIT:-50}"
|
||||
|
||||
run_query() {
|
||||
local query="$1"
|
||||
|
||||
if docker ps --format '{{.Names}}' | grep -Fxq "$CLICKHOUSE_CONTAINER"; then
|
||||
docker exec -i "$CLICKHOUSE_CONTAINER" clickhouse-client \
|
||||
--database "$CLICKHOUSE_DATABASE" --query "$query"
|
||||
return
|
||||
fi
|
||||
|
||||
"$CLICKHOUSE_CLIENT_BIN" --database "$CLICKHOUSE_DATABASE" --query "$query"
|
||||
}
|
||||
|
||||
cd "$ROOT_DIR"
|
||||
|
||||
printf '\n[unknown subjects]\n'
|
||||
run_query "
|
||||
SELECT *
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_unknown_subjects
|
||||
LIMIT $LIMIT
|
||||
FORMAT PrettyCompact
|
||||
"
|
||||
|
||||
printf '\n[unknown processes]\n'
|
||||
run_query "
|
||||
SELECT *
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_unknown_processes
|
||||
LIMIT $LIMIT
|
||||
FORMAT PrettyCompact
|
||||
"
|
||||
|
||||
printf '\n[unknown domains]\n'
|
||||
run_query "
|
||||
SELECT *
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_unknown_domains
|
||||
LIMIT $LIMIT
|
||||
FORMAT PrettyCompact
|
||||
"
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
|
||||
CLICKHOUSE_CONTAINER="${CLICKHOUSE_CONTAINER:-aw-rus-workforce-clickhouse}"
|
||||
CLICKHOUSE_DATABASE="${CLICKHOUSE_DATABASE:-aw_workforce}"
|
||||
CLICKHOUSE_USER="${CLICKHOUSE_USER:-}"
|
||||
CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-}"
|
||||
CLICKHOUSE_CLIENT_BIN="${CLICKHOUSE_CLIENT_BIN:-clickhouse-client}"
|
||||
LOAD_DEMO_SEED="${LOAD_DEMO_SEED:-1}"
|
||||
CLICKHOUSE_READY_TIMEOUT_SEC="${CLICKHOUSE_READY_TIMEOUT_SEC:-60}"
|
||||
|
||||
client_auth_args=()
|
||||
if [[ -n "$CLICKHOUSE_USER" ]]; then
|
||||
client_auth_args+=(--user "$CLICKHOUSE_USER")
|
||||
fi
|
||||
if [[ -n "$CLICKHOUSE_PASSWORD" ]]; then
|
||||
client_auth_args+=(--password "$CLICKHOUSE_PASSWORD")
|
||||
fi
|
||||
|
||||
run_client() {
|
||||
local query_file="${1:-}"
|
||||
local query="${2:-}"
|
||||
|
||||
if docker ps --format '{{.Names}}' | grep -Fxq "$CLICKHOUSE_CONTAINER"; then
|
||||
if [[ -n "$query_file" ]]; then
|
||||
docker exec -i "$CLICKHOUSE_CONTAINER" clickhouse-client "${client_auth_args[@]}" --multiquery <"$query_file"
|
||||
else
|
||||
docker exec -i "$CLICKHOUSE_CONTAINER" clickhouse-client "${client_auth_args[@]}" --database "$CLICKHOUSE_DATABASE" --query "$query"
|
||||
fi
|
||||
return
|
||||
fi
|
||||
|
||||
if command -v "$CLICKHOUSE_CLIENT_BIN" >/dev/null 2>&1; then
|
||||
if [[ -n "$query_file" ]]; then
|
||||
"$CLICKHOUSE_CLIENT_BIN" "${client_auth_args[@]}" --multiquery <"$query_file"
|
||||
else
|
||||
"$CLICKHOUSE_CLIENT_BIN" "${client_auth_args[@]}" --database "$CLICKHOUSE_DATABASE" --query "$query"
|
||||
fi
|
||||
return
|
||||
fi
|
||||
|
||||
printf 'No running ClickHouse container "%s" and no %s in PATH\n' \
|
||||
"$CLICKHOUSE_CONTAINER" "$CLICKHOUSE_CLIENT_BIN" >&2
|
||||
return 127
|
||||
}
|
||||
|
||||
wait_for_clickhouse() {
|
||||
local deadline
|
||||
deadline=$((SECONDS + CLICKHOUSE_READY_TIMEOUT_SEC))
|
||||
while (( SECONDS < deadline )); do
|
||||
if run_client "" "SELECT 1" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
printf 'ClickHouse is not ready after %s seconds\n' "$CLICKHOUSE_READY_TIMEOUT_SEC" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
apply_sql_dir() {
|
||||
local sql_file
|
||||
for sql_file in "$ROOT_DIR"/clickhouse/init/*.sql; do
|
||||
printf '[sql] %s\n' "${sql_file#$ROOT_DIR/}"
|
||||
run_client "$sql_file" ""
|
||||
done
|
||||
}
|
||||
|
||||
assert_scalar_nonzero() {
|
||||
local name="$1"
|
||||
local query="$2"
|
||||
local value
|
||||
value="$(run_client "" "$query" | tr -d '[:space:]')"
|
||||
if [[ ! "$value" =~ ^[0-9]+$ ]] || (( value < 1 )); then
|
||||
printf '[FAIL] %s: expected positive integer, got "%s"\n' "$name" "$value" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '[OK] %s: %s\n' "$name" "$value"
|
||||
}
|
||||
|
||||
assert_no_dictionary_errors() {
|
||||
local errors
|
||||
errors="$(run_client "" "
|
||||
SELECT count()
|
||||
FROM system.dictionaries
|
||||
WHERE database = '$CLICKHOUSE_DATABASE'
|
||||
AND name IN ('dict_workstation_user', 'dict_application_category', 'dict_domain_category')
|
||||
AND (status != 'LOADED' OR last_exception != '')
|
||||
")"
|
||||
errors="$(printf '%s' "$errors" | tr -d '[:space:]')"
|
||||
if [[ "$errors" != "0" ]]; then
|
||||
printf '[FAIL] dictionaries have load errors\n' >&2
|
||||
run_client "" "
|
||||
SELECT database, name, status, last_exception
|
||||
FROM system.dictionaries
|
||||
WHERE database = '$CLICKHOUSE_DATABASE'
|
||||
AND name IN ('dict_workstation_user', 'dict_application_category', 'dict_domain_category')
|
||||
FORMAT Vertical
|
||||
"
|
||||
return 1
|
||||
fi
|
||||
printf '[OK] dictionaries loaded\n'
|
||||
}
|
||||
|
||||
wait_for_clickhouse
|
||||
apply_sql_dir
|
||||
|
||||
if [[ "$LOAD_DEMO_SEED" == "1" ]]; then
|
||||
printf '[sql] sample/seed_demo.sql\n'
|
||||
run_client "$ROOT_DIR/sample/seed_demo.sql" ""
|
||||
fi
|
||||
|
||||
assert_no_dictionary_errors
|
||||
assert_scalar_nonzero "dictionary count" "
|
||||
SELECT count()
|
||||
FROM system.dictionaries
|
||||
WHERE database = '$CLICKHOUSE_DATABASE'
|
||||
AND name IN ('dict_workstation_user', 'dict_application_category', 'dict_domain_category')
|
||||
"
|
||||
assert_scalar_nonzero "hourly aggregate rows" "
|
||||
SELECT count()
|
||||
FROM $CLICKHOUSE_DATABASE.agg_workforce_productivity_hourly
|
||||
"
|
||||
assert_scalar_nonzero "daily productivity rows" "
|
||||
SELECT count()
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_productivity_daily
|
||||
"
|
||||
assert_scalar_nonzero "unknown quality rows" "
|
||||
SELECT count()
|
||||
FROM $CLICKHOUSE_DATABASE.v_workforce_unknown_quality_daily
|
||||
"
|
||||
|
||||
printf '[OK] ClickHouse workforce smoke completed\n'
|
||||
@@ -0,0 +1,37 @@
|
||||
INSERT INTO aw_workforce.dim_workstation_user
|
||||
(host_name, user_login, user_domain, employee_id, employee_name, department, branch, position, source)
|
||||
VALUES
|
||||
('ws-001', 'ivanov', 'corp', 'E001', 'Иванов И.И.', 'Бухгалтерия', 'Филиал 1', 'Бухгалтер', 'demo'),
|
||||
('ws-002', 'petrova', 'corp', 'E002', 'Петрова П.П.', 'Операционный отдел', 'Филиал 1', 'Оператор', 'demo');
|
||||
|
||||
INSERT INTO aw_workforce.dim_application_category
|
||||
(process_name, application_name, vendor, category, productivity_class, risk_level, is_system, source, comment)
|
||||
VALUES
|
||||
('1cv8.exe', '1C:Enterprise', '1C', '1c', 'productive', 'low', 0, 'demo', 'core business app'),
|
||||
('chrome.exe', 'Google Chrome', 'Google', 'browser', 'neutral', 'low', 0, 'demo', 'domain classified separately'),
|
||||
('soffice.bin', 'LibreOffice', 'The Document Foundation', 'office', 'productive', 'low', 0, 'demo', 'office suite');
|
||||
|
||||
INSERT INTO aw_workforce.dim_domain_category
|
||||
(domain, site_name, category, productivity_class, risk_level, business_allowed, source, comment)
|
||||
VALUES
|
||||
('intranet.local', 'Internal portal', 'internal_service', 'productive', 'low', 1, 'demo', 'internal work portal'),
|
||||
('github.com', 'GitHub', 'developer', 'productive', 'low', 1, 'demo', 'developer workflow'),
|
||||
('youtube.com', 'YouTube', 'media', 'neutral', 'medium', 0, 'demo', 'context-dependent media');
|
||||
|
||||
SYSTEM RELOAD DICTIONARY aw_workforce.dict_workstation_user;
|
||||
SYSTEM RELOAD DICTIONARY aw_workforce.dict_application_category;
|
||||
SYSTEM RELOAD DICTIONARY aw_workforce.dict_domain_category;
|
||||
|
||||
INSERT INTO aw_workforce.aw_window_events
|
||||
(event_time, host_name, user_login, process_name, window_title, duration_sec, source_bucket, source_event_id)
|
||||
VALUES
|
||||
(now() - INTERVAL 20 MINUTE, 'ws-001', 'ivanov', '1cv8.exe', '1C - документы', 900, 'demo-window', 'w-001'),
|
||||
(now() - INTERVAL 15 MINUTE, 'ws-002', 'petrova', 'soffice.bin', 'Отчет', 600, 'demo-window', 'w-002'),
|
||||
(now() - INTERVAL 10 MINUTE, 'ws-unknown', 'unknown', 'unknown.exe', 'Unknown tool', 120, 'demo-window', 'w-003');
|
||||
|
||||
INSERT INTO aw_workforce.aw_browser_events
|
||||
(event_time, host_name, user_login, browser_name, url, title, duration_sec, source_bucket, source_event_id)
|
||||
VALUES
|
||||
(now() - INTERVAL 9 MINUTE, 'ws-001', 'ivanov', 'chrome.exe', 'https://intranet.local/tasks', 'Tasks', 300, 'demo-browser', 'b-001'),
|
||||
(now() - INTERVAL 8 MINUTE, 'ws-002', 'petrova', 'chrome.exe', 'https://github.com/igor04091968/AWatch-rus', 'AWatch-rus', 240, 'demo-browser', 'b-002'),
|
||||
(now() - INTERVAL 7 MINUTE, 'ws-002', 'petrova', 'chrome.exe', 'https://unknown.example/path', 'Unknown', 90, 'demo-browser', 'b-003');
|
||||
@@ -0,0 +1,6 @@
|
||||
INSERT INTO aw_workforce.dim_workstation_user
|
||||
(host_name, user_login, user_domain, employee_id, employee_name, department, branch, position, source)
|
||||
VALUES
|
||||
('SHARKON2025', 'user1', 'sharkon2025', 'sharkon2025\\user1', 'user1', 'tsj', 'tsj', 'RDP user', 'manual-p3');
|
||||
|
||||
SYSTEM RELOAD DICTIONARY aw_workforce.dict_workstation_user;
|
||||
Reference in New Issue
Block a user