Compare commits

..

4 Commits

Author SHA1 Message Date
Alexander Petric
387a0c854d fix: pipeline RESET ALL with first query via tokio::join
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-03 20:41:48 +00:00
Alexander Petric
43d2cf4522 fix: move RESET ALL inside result_f async block
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-03 20:33:13 +00:00
Alexander Petric
3f55f226c3 Merge branch 'main' into alp/postgres_state_reset 2026-04-03 16:02:37 -04:00
Alexander Petric
1904ffcaaa fix: reset postgres session state on cached connection reuse
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-03 20:00:21 +00:00
219 changed files with 4141 additions and 15470 deletions

View File

@@ -1,128 +1,5 @@
# Changelog
## [1.679.0](https://github.com/windmill-labs/windmill/compare/v1.678.0...v1.679.0) (2026-04-07)
### Features
* Fork datatables ([#8339](https://github.com/windmill-labs/windmill/issues/8339)) ([3d4f4c6](https://github.com/windmill-labs/windmill/commit/3d4f4c6c38155396e9b2236a6a7a7ad4e02da877))
## [1.678.0](https://github.com/windmill-labs/windmill/compare/v1.677.0...v1.678.0) (2026-04-07)
### Features
* accept any content type on webhooks/http triggers with fallback ([#8743](https://github.com/windmill-labs/windmill/issues/8743)) ([208a597](https://github.com/windmill-labs/windmill/commit/208a597d599b4d203f7ab817a5d8ce2c06f79d0a))
* add download all logs button for flow jobs ([#8748](https://github.com/windmill-labs/windmill/issues/8748)) ([d938625](https://github.com/windmill-labs/windmill/commit/d938625785ba301fbd2c5f3d001c320eab1c504c))
### Bug Fixes
* delete raw_script_temp rows before workspace deletion to avoid FK violation ([#8752](https://github.com/windmill-labs/windmill/issues/8752)) ([8b9523e](https://github.com/windmill-labs/windmill/commit/8b9523e03c82c5a095b7cb2d5f70a87b7bbc8608))
* Fix FlowTimeline duplicate key ([#8754](https://github.com/windmill-labs/windmill/issues/8754)) ([2413dbe](https://github.com/windmill-labs/windmill/commit/2413dbefe3cc3b65c28bea437cd4471cf7e9ecba))
* remove span.enter() in dedicated worker to prevent tracing panic ([#8749](https://github.com/windmill-labs/windmill/issues/8749)) ([db55e8e](https://github.com/windmill-labs/windmill/commit/db55e8efb0c9ae198ca5ac7013439a94dfe9f550))
* restore ai agent tool deletion ([#8744](https://github.com/windmill-labs/windmill/issues/8744)) ([2f7ba9e](https://github.com/windmill-labs/windmill/commit/2f7ba9edac1a57dfc0eb3417574c72292855fc56))
## [1.677.0](https://github.com/windmill-labs/windmill/compare/v1.676.0...v1.677.0) (2026-04-06)
### Features
* add AWS Secrets Manager as secret storage backend (Beta) ([#8734](https://github.com/windmill-labs/windmill/issues/8734)) ([09bbc18](https://github.com/windmill-labs/windmill/commit/09bbc18bb773d9ffaa5aaa4bd9d7ce296f3ac468))
### Bug Fixes
* remove stale KMS openapi/description, restore stripped doc comments ([c09a431](https://github.com/windmill-labs/windmill/commit/c09a4311fd73c58acc8f3997428f002598dacce6))
* use runnable key for file naming in generate-metadata to prevent duplicate scripts in raw apps ([#8740](https://github.com/windmill-labs/windmill/issues/8740)) ([edfe074](https://github.com/windmill-labs/windmill/commit/edfe074e98cb3955be0768de7ed19e6ed8525916))
## [1.676.0](https://github.com/windmill-labs/windmill/compare/v1.675.1...v1.676.0) (2026-04-06)
### Features
* add path name autocomplete with ghost text and folder cycling ([#8731](https://github.com/windmill-labs/windmill/issues/8731)) ([e326621](https://github.com/windmill-labs/windmill/commit/e32662169a9762605de2dbe058514ddefbe07982))
### Bug Fixes
* fix custom urls not found ([d2abc0d](https://github.com/windmill-labs/windmill/commit/d2abc0d4300bb53f4035102f214d3c05bf0976a1))
### Performance Improvements
* add partial index for expired cache resource cleanup ([#8728](https://github.com/windmill-labs/windmill/issues/8728)) ([c721fac](https://github.com/windmill-labs/windmill/commit/c721fac466524747de04e3623c8cd62de8bd4dae))
## [1.675.1](https://github.com/windmill-labs/windmill/compare/v1.675.0...v1.675.1) (2026-04-05)
### Bug Fixes
* log cleanup scans S3 orphans and works cross-server ([#8729](https://github.com/windmill-labs/windmill/issues/8729)) ([f703fba](https://github.com/windmill-labs/windmill/commit/f703fba1ef56c89a97b2b4da7b4c188158f4c982))
### Performance Improvements
* add indexes for cleanup deletes on concurrency_key and autoscaling_event ([#8726](https://github.com/windmill-labs/windmill/issues/8726)) ([eae46a2](https://github.com/windmill-labs/windmill/commit/eae46a21a93fe7ab191228658dd5825f472bd851))
## [1.675.0](https://github.com/windmill-labs/windmill/compare/v1.674.2...v1.675.0) (2026-04-05)
### Features
* add object storage usage view and manual log cleanup ([#8724](https://github.com/windmill-labs/windmill/issues/8724)) ([02d0ee9](https://github.com/windmill-labs/windmill/commit/02d0ee919880823a33b112bcaf626a8933e1f715))
### Bug Fixes
* add admin check to count_completed_jobs_detail and document query builder SQL safety ([#8722](https://github.com/windmill-labs/windmill/issues/8722)) ([dd39c11](https://github.com/windmill-labs/windmill/commit/dd39c110a8468bf31d42428fc978cd302426fa86))
* allow private AI base URLs in ai_proxy integration test ([#8715](https://github.com/windmill-labs/windmill/issues/8715)) ([2b865c0](https://github.com/windmill-labs/windmill/commit/2b865c0694d79ce6477e5f14a077b73837007500))
* enrich OTEL spans with job_kind, trigger_kind, trigger, created_by, and script_hash ([#8718](https://github.com/windmill-labs/windmill/issues/8718)) ([7bf6ac2](https://github.com/windmill-labs/windmill/commit/7bf6ac2b694fc829327248ff2480c20c97e03e48))
* split DB health endpoint and add slow query controls ([#8725](https://github.com/windmill-labs/windmill/issues/8725)) ([01e39d9](https://github.com/windmill-labs/windmill/commit/01e39d9cd1b841d085bcc28a578654a5486cf76e))
## [1.674.2](https://github.com/windmill-labs/windmill/compare/v1.674.1...v1.674.2) (2026-04-04)
### Bug Fixes
* enforce RLS on $var: resolution in AI proxy (GHSA-jwg4-v3cj-rvfm) ([#8713](https://github.com/windmill-labs/windmill/issues/8713)) ([ff8e39c](https://github.com/windmill-labs/windmill/commit/ff8e39c69b1438defcaabd9d4906e7adafa7010c))
* SSRF via X-Resource-Path header in AI proxy endpoint ([#8712](https://github.com/windmill-labs/windmill/issues/8712)) ([f394e67](https://github.com/windmill-labs/windmill/commit/f394e674f22af13bb77915f33aa1e8de402b6fe1))
## [1.674.1](https://github.com/windmill-labs/windmill/compare/v1.674.0...v1.674.1) (2026-04-04)
### Bug Fixes
* create pg connection for cloud-hosted jobs instead of panicking ([#8710](https://github.com/windmill-labs/windmill/issues/8710)) ([aff95c3](https://github.com/windmill-labs/windmill/commit/aff95c33b2fd4c248dfaf595b8d18a6dbc50f0e6))
## [1.674.0](https://github.com/windmill-labs/windmill/compare/v1.673.0...v1.674.0) (2026-04-03)
### Features
* add application-level heartbeat support for websocket triggers ([#8686](https://github.com/windmill-labs/windmill/issues/8686)) ([5b7fa63](https://github.com/windmill-labs/windmill/commit/5b7fa63bf1800313e9b82465b8a4399a48634371))
* add Azure Key Vault as secret storage backend ([#8704](https://github.com/windmill-labs/windmill/issues/8704)) ([dcd615f](https://github.com/windmill-labs/windmill/commit/dcd615fdc3c66ec2a8e39c01f8142a7e7c82f534))
* add http/protobuf support for OTEL exporters ([#8702](https://github.com/windmill-labs/windmill/issues/8702)) ([0aea49f](https://github.com/windmill-labs/windmill/commit/0aea49f9607d5cbb5bcfa3068a179c9b7bf9afd6))
* add optional labels to scripts, flows, apps, schedules, triggers ([#8609](https://github.com/windmill-labs/windmill/issues/8609)) ([c4c9ef5](https://github.com/windmill-labs/windmill/commit/c4c9ef5fd7b41052b08ee941725434e8ca4ac970))
* add powershell common parameters support ([#8683](https://github.com/windmill-labs/windmill/issues/8683)) ([0317d58](https://github.com/windmill-labs/windmill/commit/0317d5891cfcfbde7b04795c034c088e933ee3d0))
* sql.raw in Typescript client ([#8706](https://github.com/windmill-labs/windmill/issues/8706)) ([ce290f6](https://github.com/windmill-labs/windmill/commit/ce290f68db866c07b30c97c2c0b3e39fee0a26d8))
* Support .ducklake() and .datatable() in agent workers ([#8697](https://github.com/windmill-labs/windmill/issues/8697)) ([fda68a7](https://github.com/windmill-labs/windmill/commit/fda68a72e5dfcded2350d1ff33ca4c695ab337b7))
### Bug Fixes
* add secretKeyRef support for jwt_secret and rsa_keys ([#8698](https://github.com/windmill-labs/windmill/issues/8698)) ([ba21470](https://github.com/windmill-labs/windmill/commit/ba214709b94f9467738e66b016331e97ac7d5d10))
* align script push metadata warning with generated locks ([#8690](https://github.com/windmill-labs/windmill/issues/8690)) ([6656b46](https://github.com/windmill-labs/windmill/commit/6656b46f10408e1c15961a72cde4c13b5c5b3923))
* debounce S3 proxy logs ([#8694](https://github.com/windmill-labs/windmill/issues/8694)) ([a3073ad](https://github.com/windmill-labs/windmill/commit/a3073ad8244efd9043e27f6731f7b53dbda662c1))
* dedicated worker dispatch, cross-workspace deps, UI improvements ([#8689](https://github.com/windmill-labs/windmill/issues/8689)) ([bffa61e](https://github.com/windmill-labs/windmill/commit/bffa61e33f2305bbeb79a2c91989a47baa7dff31))
* gate relock_skip tests on private feature and update ee-repo-ref ([#8703](https://github.com/windmill-labs/windmill/issues/8703)) ([adc9fe7](https://github.com/windmill-labs/windmill/commit/adc9fe722d8511a5914d81faac40af757e7f5e3f))
* hide deprecated cli metadata commands ([#8699](https://github.com/windmill-labs/windmill/issues/8699)) ([b960598](https://github.com/windmill-labs/windmill/commit/b96059843168c072f24072f93fecd80431e5d4cf))
* optimize S3 proxy performance ([#8685](https://github.com/windmill-labs/windmill/issues/8685)) ([0cfa462](https://github.com/windmill-labs/windmill/commit/0cfa462c379e887fdb5ad5e3bbff7798648d4e91))
* pipeline DISCARD ALL with first query on cached pg connections ([#8707](https://github.com/windmill-labs/windmill/issues/8707)) ([6d58d1a](https://github.com/windmill-labs/windmill/commit/6d58d1a74d1e69b163210a795502a7b3931001b5))
* resolve schedule update deadlock ([#8701](https://github.com/windmill-labs/windmill/issues/8701)) ([27ca417](https://github.com/windmill-labs/windmill/commit/27ca417201c99cf6fe0ae5b52a63c0395033e196))
* support raw app deployment history ([#8657](https://github.com/windmill-labs/windmill/issues/8657)) ([f234df9](https://github.com/windmill-labs/windmill/commit/f234df97ec3cdc480ee9d403370a3512496b024b))
* use pre-aggregated stats for telemetry job usage queries ([#8688](https://github.com/windmill-labs/windmill/issues/8688)) ([cdf3c29](https://github.com/windmill-labs/windmill/commit/cdf3c29664e4142c0f4487c07e585d1af3f97f91))
## [1.673.0](https://github.com/windmill-labs/windmill/compare/v1.672.0...v1.673.0) (2026-04-02)

View File

@@ -162,19 +162,11 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH
RUN apt-get update \
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini \
&& if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository
RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends postgresql-client \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN if [ "$WITH_GIT" = "true" ]; then \
apt-get update -y \
&& apt-get install -y git \

View File

@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status\n SET flow_status = JSONB_SET(flow_status, ARRAY['cleanup_module', 'flow_jobs_to_clean'], COALESCE(flow_status->'cleanup_module'->'flow_jobs_to_clean', '[]'::jsonb) || $1)\n WHERE id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb",
"Uuid"
]
},
"nullable": []
},
"hash": "007fa93171b244490b94464938b9f95aca4e91bccde6da93cb151799b3398049"
}

View File

@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE background_task_state\n SET value = $1, updated_at = NOW()\n WHERE name = $2 AND owner = $3 AND running = true",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "00bdac005e3fb2167c90c26b96ab267c19d4b6578736c852114df96ed5575952"
}

View File

@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO job_delete_schedule (job_id, workspace_id, delete_at) VALUES ($1, $2, now() + make_interval(secs => $3::double precision)) ON CONFLICT (job_id) DO NOTHING",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Float8"
]
},
"nullable": []
},
"hash": "035e29e775bfc5b236100135e1d94a4baf2b617b86f0c3c74ba9a00b859993f6"
}

View File

@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO script\n (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n FROM script WHERE hash = $2 AND workspace_id = $3;\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Int8",
"Text"
]
},
"nullable": []
},
"hash": "0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034"
}

View File

@@ -1,14 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM raw_script_temp WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "1325d08f0d34a5e706543585186dac4ee099aefcd3ed51f10f3298a2c006da80"
}

View File

@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status\n SET flow_status = JSONB_SET(flow_status, ARRAY['cleanup_module', 'flow_jobs_to_schedule_clean'], COALESCE(flow_status->'cleanup_module'->'flow_jobs_to_schedule_clean', '[]'::jsonb) || $1)\n WHERE id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb",
"Uuid"
]
},
"nullable": []
},
"hash": "21331baf02c3c798bcc215443dad16eb66acac5f40c20529b595154f4e6fd754"
}

View File

@@ -1,26 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT workspace_id, path FROM variable WHERE is_secret = true AND value LIKE '$aws_sm:%'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
false
]
},
"hash": "243588f12c62aa7913bc3a4ae11ecee8a1e03863735036b1192fca143f27641c"
}

View File

@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO script (\n workspace_id, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n )\n SELECT\n $1, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n FROM script\n WHERE workspace_id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text"
]
},
"nullable": []
},
"hash": "2c256552a430877c42224055aeb81df33d88ff295483cb28369eda42ce58afec"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT path, value FROM variable\n WHERE path LIKE ('u/' || $1 || '/%')\n AND workspace_id = $2\n AND is_secret = true\n AND (value LIKE '$vault:%' OR value LIKE '$azure_kv:%')",
"query": "SELECT path, value FROM variable\n WHERE path LIKE ('u/' || $1 || '/%')\n AND workspace_id = $2\n AND is_secret = true\n AND value LIKE '$vault:%'",
"describe": {
"columns": [
{
@@ -25,5 +25,5 @@
false
]
},
"hash": "bda9fc0c6e41d8afebee538e6997c6bad7843f93d0a5780632b02fa215aa404b"
"hash": "2fcddda99dd0aacf5007ed459cb27caa754424e062427edf5ddcb95f9d96888e"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n table_name as \"table_name!\",\n SUM(live_tuples)::bigint as \"live_tuples!\",\n SUM(dead_tuples)::bigint as \"dead_tuples!\",\n MAX(last_autovacuum) as \"last_autovacuum\",\n MAX(last_autoanalyze) as \"last_autoanalyze\"\n FROM (\n SELECT\n CASE\n WHEN i.inhparent IS NOT NULL THEN schemaname || '.' || p.relname\n ELSE schemaname || '.' || s.relname\n END as table_name,\n COALESCE(n_live_tup, 0) as live_tuples,\n COALESCE(n_dead_tup, 0) as dead_tuples,\n last_autovacuum,\n last_autoanalyze\n FROM pg_stat_user_tables s\n LEFT JOIN pg_class c ON c.relname = s.relname AND c.relnamespace = (\n SELECT oid FROM pg_namespace WHERE nspname = s.schemaname\n )\n LEFT JOIN pg_inherits i ON i.inhrelid = c.oid\n LEFT JOIN pg_class p ON p.oid = i.inhparent\n ) sub\n GROUP BY table_name\n HAVING SUM(live_tuples) + SUM(dead_tuples) >= 1000\n ORDER BY SUM(dead_tuples) DESC",
"query": "SELECT\n table_name as \"table_name!\",\n SUM(live_tuples)::bigint as \"live_tuples!\",\n SUM(dead_tuples)::bigint as \"dead_tuples!\",\n MAX(last_autovacuum) as \"last_autovacuum\",\n MAX(last_autoanalyze) as \"last_autoanalyze\"\n FROM (\n SELECT\n CASE\n WHEN i.inhparent IS NOT NULL THEN schemaname || '.' || p.relname\n ELSE schemaname || '.' || s.relname\n END as table_name,\n COALESCE(n_live_tup, 0) as live_tuples,\n COALESCE(n_dead_tup, 0) as dead_tuples,\n last_autovacuum,\n last_autoanalyze\n FROM pg_stat_user_tables s\n LEFT JOIN pg_class c ON c.relname = s.relname AND c.relnamespace = (\n SELECT oid FROM pg_namespace WHERE nspname = s.schemaname\n )\n LEFT JOIN pg_inherits i ON i.inhrelid = c.oid\n LEFT JOIN pg_class p ON p.oid = i.inhparent\n ) sub\n GROUP BY table_name\n ORDER BY SUM(dead_tuples) DESC",
"describe": {
"columns": [
{
@@ -40,5 +40,5 @@
null
]
},
"hash": "a5455608f60ad3939b337e374cbb2f4dbf98db86864e58fef47447390bb5fcac"
"hash": "30930bfb0513f1a70194a900011b2e890bc4146bb0419210cd76743cacda8bfa"
}

View File

@@ -50,36 +50,31 @@
},
{
"ordinal": 9,
"name": "ws_specific",
"type_info": "Bool"
},
{
"ordinal": 10,
"name": "is_expired",
"type_info": "Bool"
},
{
"ordinal": 11,
"ordinal": 10,
"name": "is_refreshed",
"type_info": "Bool"
},
{
"ordinal": 12,
"ordinal": 11,
"name": "refresh_error",
"type_info": "Text"
},
{
"ordinal": 13,
"ordinal": 12,
"name": "is_linked",
"type_info": "Bool"
},
{
"ordinal": 14,
"ordinal": 13,
"name": "is_oauth?",
"type_info": "Bool"
},
{
"ordinal": 15,
"ordinal": 14,
"name": "account",
"type_info": "Int4"
}
@@ -100,7 +95,6 @@
true,
true,
true,
false,
null,
null,
true,

View File

@@ -47,11 +47,6 @@
"ordinal": 8,
"name": "labels",
"type_info": "TextArray"
},
{
"ordinal": 9,
"name": "ws_specific",
"type_info": "Bool"
}
],
"parameters": {
@@ -68,8 +63,7 @@
false,
true,
true,
true,
false
true
]
},
"hash": "45e4d13f5806122faecdb1d9ab18159555b652869a036b006f4a151e999b17b7"

View File

@@ -1,21 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO resource\n (workspace_id, path, value, description, resource_type, created_by, edited_at, ws_specific, labels)\n VALUES ($1, $2, $3, $4, $5, $6, now(), $7, $8) ON CONFLICT (workspace_id, path)\n DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description, resource_type = EXCLUDED.resource_type, edited_at = now(), ws_specific = EXCLUDED.ws_specific, labels = EXCLUDED.labels",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Jsonb",
"Text",
"Varchar",
"Varchar",
"Bool",
"TextArray"
]
},
"nullable": []
},
"hash": "4a1947f86ab5d2ff63a3f307c6f537caaf6d9177d72f1deeab4b5afcb83c0547"
}

View File

@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT datatable->'datatables' FROM workspace_settings WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "4d03f08c86588d3fba178340a3b64f375f82e8904920d51454f10e04fc24b62a"
}

View File

@@ -1,24 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT created_by FROM v2_job WHERE id = $1 AND workspace_id = $2 AND ($3::text[] IS NULL OR tag = ANY($3))",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "created_by",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Uuid",
"Text",
"TextArray"
]
},
"nullable": [
false
]
},
"hash": "56cb2ff34e93ea7ebe9e3b4c98f086793ba0ad2ad00c4361fef4f183557c50fd"
}

View File

@@ -1,83 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "WITH RECURSIVE job_tree AS (\n SELECT j.id, j.kind::text, j.flow_step_id, j.parent_job,\n '' as path_label, 0 as depth,\n j.id::text as id_path,\n ''::text as parent_module_type\n FROM v2_job j\n WHERE j.id = $2 AND j.workspace_id = $1\n UNION ALL\n SELECT j.id, j.kind::text, j.flow_step_id, j.parent_job,\n CASE\n WHEN jt.path_label = '' THEN COALESCE(j.flow_step_id, '')\n ELSE jt.path_label || '/' || COALESCE(j.flow_step_id, '')\n END,\n jt.depth + 1,\n jt.id_path || '/' || j.id::text,\n COALESCE((\n SELECT m->'value'->>'type'\n FROM v2_job parent_j\n LEFT JOIN flow f ON f.path = parent_j.runnable_path\n AND f.workspace_id = parent_j.workspace_id\n LEFT JOIN flow_node fn ON fn.id = parent_j.runnable_id\n CROSS JOIN LATERAL jsonb_array_elements(\n COALESCE(parent_j.raw_flow, f.value, fn.flow)->'modules'\n ) m\n WHERE parent_j.id = jt.id\n AND m->>'id' = j.flow_step_id\n LIMIT 1\n ), '')::text\n FROM v2_job j\n JOIN job_tree jt ON j.parent_job = jt.id\n WHERE j.workspace_id = $1\n ),\n with_sibling_index AS (\n SELECT jt.*,\n ROW_NUMBER() OVER (\n PARTITION BY jt.parent_job, jt.flow_step_id\n ORDER BY jt.id\n ) as sibling_index,\n COUNT(*) OVER (\n PARTITION BY jt.parent_job, jt.flow_step_id\n ) as sibling_count\n FROM job_tree jt\n )\n SELECT w.id, w.kind, w.flow_step_id, w.path_label,\n w.sibling_index::int as sibling_index,\n w.sibling_count::int as sibling_count,\n w.depth::int as depth,\n w.parent_module_type,\n coalesce(job_logs.logs, '') as logs,\n COALESCE(job_logs.log_offset, 0) as log_offset,\n job_logs.log_file_index\n FROM with_sibling_index w\n LEFT JOIN job_logs ON job_logs.job_id = w.id\n ORDER BY w.id_path ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "kind",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "flow_step_id",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "path_label",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "sibling_index",
"type_info": "Int4"
},
{
"ordinal": 5,
"name": "sibling_count",
"type_info": "Int4"
},
{
"ordinal": 6,
"name": "depth",
"type_info": "Int4"
},
{
"ordinal": 7,
"name": "parent_module_type",
"type_info": "Text"
},
{
"ordinal": 8,
"name": "logs",
"type_info": "Text"
},
{
"ordinal": 9,
"name": "log_offset",
"type_info": "Int4"
},
{
"ordinal": 10,
"name": "log_file_index",
"type_info": "TextArray"
}
],
"parameters": {
"Left": [
"Text",
"Uuid"
]
},
"nullable": [
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
true
]
},
"hash": "5b95f464d492227ffe8a449844da00a6539788f9be647d6b52f01cbb2967473e"
}

View File

@@ -1,25 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, true, $3, NOW(), NULL, NOW())\n ON CONFLICT (name) DO UPDATE SET\n value = EXCLUDED.value,\n running = true,\n owner = EXCLUDED.owner,\n started_at = NOW(),\n finished_at = NULL,\n updated_at = NOW()\n WHERE background_task_state.running = false\n OR background_task_state.updated_at < NOW() - ($4::text)::interval\n RETURNING 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text",
"Jsonb",
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "5bd1fd2d4f2104020aecdaaf4fc96af9a6a84e6dfe9418d2f8c8a362ed56cc06"
}

View File

@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id FROM v2_job WHERE id = ANY($1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"UuidArray"
]
},
"nullable": [
false
]
},
"hash": "63a1d389bcc923c0e2e3dc9266097ff1c236f0af375f0d73fa9e11ad351d6dd6"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41)",
"query": "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40)",
"describe": {
"columns": [],
"parameters": {
@@ -77,7 +77,6 @@
"Bool",
"Bool",
"Int4",
"Int4",
"Varchar",
"Bool",
"Varchar",
@@ -96,5 +95,5 @@
},
"nullable": []
},
"hash": "dafc503a5f3adc5c7db7c11096775cacfffd2d3173dfdddcf37589cba356791e"
"hash": "790d79ec7abe6ebe1092afd9de4c5fc383272d2057ac5b47a7425f095f4e8788"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT value, description, resource_type, ws_specific\n FROM resource\n WHERE workspace_id = $1 AND path = $2",
"query": "SELECT value, description, resource_type\n FROM resource\n WHERE workspace_id = $1 AND path = $2",
"describe": {
"columns": [
{
@@ -17,11 +17,6 @@
"ordinal": 2,
"name": "resource_type",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "ws_specific",
"type_info": "Bool"
}
],
"parameters": {
@@ -33,9 +28,8 @@
"nullable": [
true,
true,
false,
false
]
},
"hash": "83d537bb8a4bb0121b6c7c2c6f7ce67c1281e16025aa3cf559a4f7a6a034d734"
"hash": "819c233915383e89af1bcf1a56c5f67c4e1fc217f216f609e36a9944a7807b33"
}

View File

@@ -1,14 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE job_logs SET logs = '##DELETED##' WHERE job_id = ANY($1)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"UuidArray"
]
},
"nullable": []
},
"hash": "954e832d2587506a4bb0cb3a4fb45658026de7da4680eca1ea6b08f4b5e33800"
}

View File

@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id FROM v2_job WHERE root_job = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
false
]
},
"hash": "9c5f9fc1c2fdf35e98c78180a319f2cfdfe09b4eef3375a6170454d3d52e8dbc"
}

View File

@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status\n SET flow_status = JSONB_SET(flow_status, ARRAY['cleanup_module', 'flow_jobs_to_schedule_clean'], COALESCE(flow_status->'cleanup_module'->'flow_jobs_to_schedule_clean', '[]'::jsonb) || $1)\n WHERE id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb",
"Uuid"
]
},
"nullable": []
},
"hash": "a25200e046d6e15bf5a5f81d6cc9622bdb7ad8009fa6e67606296c1a8a1a92b4"
}

View File

@@ -1,26 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT workspace_id, path FROM variable WHERE is_secret = true AND value LIKE '$azure_kv:%'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
false
]
},
"hash": "a450c5d5fe4bc15e13ea389fadd21a769b029b650a334e48bb3198c049bb92fe"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT value, is_secret\n FROM variable\n WHERE path = $1 AND workspace_id = $2",
"query": "SELECT value, is_secret \n FROM variable \n WHERE path = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
@@ -25,5 +25,5 @@
false
]
},
"hash": "61f3627cbd2a50b745c4e491ed8ee71e58db22b7f53202c5360a01c92091fc08"
"hash": "a5a03b9235b25bca359235f2e546197f02ca1cf898d7f2686419749b1cb0679e"
}

View File

@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO script\n (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n FROM script WHERE hash = $2 AND workspace_id = $3;\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Int8",
"Text"
]
},
"nullable": []
},
"hash": "a969194571dd3f12e628ce0f01b0ddc09bbcf4506eff2290664c295d16fec4ae"
}

View File

@@ -1,58 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, value as \"value!: serde_json::Value\", running as \"running!\",\n owner, started_at, finished_at, updated_at as \"updated_at!\"\n FROM background_task_state WHERE name = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "value!: serde_json::Value",
"type_info": "Jsonb"
},
{
"ordinal": 2,
"name": "running!",
"type_info": "Bool"
},
{
"ordinal": 3,
"name": "owner",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 5,
"name": "finished_at",
"type_info": "Timestamptz"
},
{
"ordinal": 6,
"name": "updated_at!",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false,
false,
true,
true,
true,
false
]
},
"hash": "aaa6f3afc5774dc5320fe68b80a58e99b0f99caca8d922ccb93aaf70014e4277"
}

View File

@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status\n SET flow_status = JSONB_SET(flow_status, ARRAY['cleanup_module', 'flow_jobs_to_clean'], COALESCE(flow_status->'cleanup_module'->'flow_jobs_to_clean', '[]'::jsonb) || $1)\n WHERE id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb",
"Uuid"
]
},
"nullable": []
},
"hash": "b01160fe44d69834ac08bbf60feacb3e3caa02a04b084da44cdcb9103794b39e"
}

View File

@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings\n SET datatable = jsonb_set(datatable, ARRAY['datatables', $2, 'forked_from'], $3::jsonb)\n WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Jsonb"
]
},
"nullable": []
},
"hash": "b2903da4186589d9df7d4e245f29138047b78f32d270a91a433cd6a04ecb48ea"
}

View File

@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*) FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
null
]
},
"hash": "b3c84299ed9872960c2287ef6e3a9dca85eeb2d22e7004021c987d08598b5585"
}

View File

@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE resource\n SET value = jsonb_set(value, '{dbname}', to_jsonb($3::text)),\n ws_specific = true\n WHERE workspace_id = $1 AND path = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "ba09bbba6bd34ef52c702eddc7b0b047e71a171cf067fcd1e812681d8bd15005"
}

View File

@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*) FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
null
]
},
"hash": "c5e72bcfd4d389fce281811d7a731711d89fb79f7b2adcfbdf33c070b3adaf7d"
}

View File

@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM log_file WHERE (file_path, hostname) IN (\n SELECT file_path, hostname FROM log_file\n WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval\n LIMIT $2\n ) RETURNING file_path, hostname",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "file_path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "hostname",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Int8",
"Int8"
]
},
"nullable": [
false,
false
]
},
"hash": "d21464c0e980b9d3a5c8f940c2abe43275843f33fb1998d9c8bd940ef127bfbf"
}

View File

@@ -1,17 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings\n SET datatable = jsonb_set(\n jsonb_set(datatable, ARRAY['datatables', $2, 'database', 'resource_path'], to_jsonb($3::text)),\n ARRAY['datatables', $2, 'forked_from'], $4::jsonb\n )\n WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text",
"Jsonb"
]
},
"nullable": []
},
"hash": "da6f37f91f2a24cf9ce8a1759adc5e9aebca35a0245598000c5d741dc6ede2d5"
}

View File

@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO resource\n (workspace_id, path, value, description, resource_type, created_by, edited_at, labels)\n VALUES ($1, $2, $3, $4, $5, $6, now(), $7) ON CONFLICT (workspace_id, path)\n DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description, resource_type = EXCLUDED.resource_type, edited_at = now(), labels = EXCLUDED.labels",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Jsonb",
"Text",
"Varchar",
"Varchar",
"TextArray"
]
},
"nullable": []
},
"hash": "deac41298e8b0d0870e314fef0813c24dd55d63bda78a0a5f35ed6f22bea6bef"
}

View File

@@ -1,14 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "df13aa5740f5475165b934f26d90b0f7c9ed4f2aa5b2d54817e7630ca1d37753"
}

View File

@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM job_delete_schedule\n WHERE job_id IN (\n SELECT job_id FROM job_delete_schedule\n WHERE delete_at <= now()\n ORDER BY delete_at\n LIMIT $1\n FOR UPDATE SKIP LOCKED\n )\n RETURNING job_id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "job_id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
false
]
},
"hash": "e1886af775f70f7ad3949e35d2e274818884b2ce38226d5d8f9b8774dda6d5dc"
}

View File

@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO script (\n workspace_id, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n )\n SELECT\n $1, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n FROM script\n WHERE workspace_id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text"
]
},
"nullable": []
},
"hash": "f0fcc355334f4d715b366e1b3be88b2b5e316536efae31da09217176b59db030"
}

View File

@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT path AS \"path!\" FROM (\n (SELECT DISTINCT path FROM script WHERE workspace_id = $1 AND archived = false AND deleted = false AND draft_only IS NOT true LIMIT 5000)\n UNION\n (SELECT path FROM flow WHERE workspace_id = $1 AND archived = false AND draft_only IS NOT true LIMIT 5000)\n UNION\n (SELECT path FROM app WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM raw_app WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM variable WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM resource WHERE workspace_id = $1 LIMIT 5000)\n ) t\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path!",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "f175f0eda0dcdb26c08b743de80e73344dff5b98a33daaee144ffaccaa8a0bad"
}

View File

@@ -1,23 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "f85208a1f14536125db503d5cf375737a5855560a7f7c4a105461af73778a078"
}

View File

@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE background_task_state\n SET value = $1, running = false, finished_at = NOW(), updated_at = NOW()\n WHERE name = $2 AND owner = $3",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "fa35db1bc957299867070a7a00da84ed2e6d319b1b7248cf99ae624cfd9fee76"
}

210
backend/Cargo.lock generated
View File

@@ -234,9 +234,9 @@ dependencies = [
[[package]]
name = "arc-swap"
version = "1.9.1"
version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207"
checksum = "a07d1f37ff60921c83bdfc7407723bdefe89b44b98a9b772f225c8f9d67141a6"
dependencies = [
"rustversion",
]
@@ -1006,30 +1006,6 @@ dependencies = [
"url",
]
[[package]]
name = "aws-sdk-secretsmanager"
version = "1.100.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26b0b2985427bb081e54e759468d3af89fa2ccb17fb8b9e5b704ae2f8da10a3b"
dependencies = [
"aws-credential-types",
"aws-runtime",
"aws-smithy-async",
"aws-smithy-http 0.63.4",
"aws-smithy-json 0.62.4",
"aws-smithy-observability",
"aws-smithy-runtime",
"aws-smithy-runtime-api",
"aws-smithy-types",
"aws-types",
"bytes",
"fastrand",
"http 0.2.12",
"http 1.4.0",
"regex-lite",
"tracing",
]
[[package]]
name = "aws-sdk-sqs"
version = "1.77.0"
@@ -2213,9 +2189,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.2.59"
version = "1.2.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7a4d3ec6524d28a329fc53654bbadc9bdd7b0431f5d65f1a56ffb28a1ee5283"
checksum = "e1e928d4b69e3077709075a938a05ffbedfa53a84c8f766efbf8220bb1ff60e1"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -5644,9 +5620,9 @@ dependencies = [
[[package]]
name = "fastrand"
version = "2.4.1"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6"
checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
[[package]]
name = "fastwebsockets"
@@ -8199,9 +8175,9 @@ dependencies = [
[[package]]
name = "libz-sys"
version = "1.1.28"
version = "1.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc3a226e576f50782b3305c5ccf458698f92798987f551c6a02efe8276721e22"
checksum = "d52f4c29e2a68ac30c9087e1b772dc9f44a2b66ed44edf2266cf2be9b03dafc1"
dependencies = [
"cc",
"libc",
@@ -9612,9 +9588,9 @@ checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
[[package]]
name = "openssl-src"
version = "300.6.0+3.6.2"
version = "300.5.5+3.5.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8e8cbfd3a4a8c8f089147fd7aaa33cf8c7450c4d09f8f80698a0cf093abeff4"
checksum = "3f1787d533e03597a7934fd0a765f0d28e94ecc5fb7789f8053b1e699a56f709"
dependencies = [
"cc",
]
@@ -11745,7 +11721,7 @@ version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver 1.0.28",
"semver 1.0.27",
]
[[package]]
@@ -12339,9 +12315,9 @@ dependencies = [
[[package]]
name = "semver"
version = "1.0.28"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"
checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2"
[[package]]
name = "semver-parser"
@@ -15764,7 +15740,7 @@ dependencies = [
"bitflags 2.9.4",
"hashbrown 0.15.5",
"indexmap 2.12.0",
"semver 1.0.28",
"semver 1.0.27",
]
[[package]]
@@ -16007,7 +15983,7 @@ dependencies = [
[[package]]
name = "windmill"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-nats",
@@ -16086,7 +16062,7 @@ dependencies = [
[[package]]
name = "windmill-alerting"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16099,7 +16075,7 @@ dependencies = [
[[package]]
name = "windmill-api"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"argon2",
@@ -16240,7 +16216,7 @@ dependencies = [
[[package]]
name = "windmill-api-agent-workers"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16263,7 +16239,7 @@ dependencies = [
[[package]]
name = "windmill-api-assets"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16276,7 +16252,7 @@ dependencies = [
[[package]]
name = "windmill-api-auth"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"axum 0.8.4",
@@ -16302,7 +16278,7 @@ dependencies = [
[[package]]
name = "windmill-api-client"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"reqwest 0.12.28",
"serde",
@@ -16312,7 +16288,7 @@ dependencies = [
[[package]]
name = "windmill-api-configs"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16329,7 +16305,7 @@ dependencies = [
[[package]]
name = "windmill-api-debug"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"base64 0.22.1",
@@ -16352,7 +16328,7 @@ dependencies = [
[[package]]
name = "windmill-api-embeddings"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"axum 0.8.4",
@@ -16375,7 +16351,7 @@ dependencies = [
[[package]]
name = "windmill-api-flow-conversations"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16391,7 +16367,7 @@ dependencies = [
[[package]]
name = "windmill-api-flows"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16411,7 +16387,7 @@ dependencies = [
[[package]]
name = "windmill-api-groups"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16431,7 +16407,7 @@ dependencies = [
[[package]]
name = "windmill-api-inputs"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16445,7 +16421,7 @@ dependencies = [
[[package]]
name = "windmill-api-integration-tests"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-nats",
@@ -16476,7 +16452,7 @@ dependencies = [
[[package]]
name = "windmill-api-jobs"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"axum 0.8.4",
@@ -16501,7 +16477,7 @@ dependencies = [
[[package]]
name = "windmill-api-npm-proxy"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"flate2",
@@ -16519,7 +16495,7 @@ dependencies = [
[[package]]
name = "windmill-api-openapi"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"axum 0.8.4",
@@ -16541,7 +16517,7 @@ dependencies = [
[[package]]
name = "windmill-api-schedule"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16561,7 +16537,7 @@ dependencies = [
[[package]]
name = "windmill-api-scripts"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16591,7 +16567,7 @@ dependencies = [
[[package]]
name = "windmill-api-settings"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"axum 0.8.4",
@@ -16618,7 +16594,7 @@ dependencies = [
[[package]]
name = "windmill-api-sse"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"lazy_static",
"serde",
@@ -16630,7 +16606,7 @@ dependencies = [
[[package]]
name = "windmill-api-users"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"argon2",
"axum 0.8.4",
@@ -16655,7 +16631,7 @@ dependencies = [
[[package]]
name = "windmill-api-workers"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16669,7 +16645,7 @@ dependencies = [
[[package]]
name = "windmill-api-workspaces"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"axum 0.8.4",
"chrono",
@@ -16696,13 +16672,12 @@ dependencies = [
"windmill-git-sync",
"windmill-object-store",
"windmill-queue",
"windmill-store",
"windmill-types",
]
[[package]]
name = "windmill-audit"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"chrono",
"lazy_static",
@@ -16716,7 +16691,7 @@ dependencies = [
[[package]]
name = "windmill-autoscaling"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"axum 0.8.4",
@@ -16735,7 +16710,7 @@ dependencies = [
[[package]]
name = "windmill-common"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"aes-gcm",
"aho-corasick",
@@ -16747,7 +16722,6 @@ dependencies = [
"aws-credential-types",
"aws-sdk-bedrockruntime",
"aws-sdk-rds",
"aws-sdk-secretsmanager",
"aws-sdk-sts",
"aws-smithy-types",
"aws-smithy-types-convert",
@@ -16801,7 +16775,7 @@ dependencies = [
"reqwest-retry",
"rsa",
"schemars 0.8.22",
"semver 1.0.28",
"semver 1.0.27",
"serde",
"serde_json",
"serde_yml",
@@ -16838,7 +16812,7 @@ dependencies = [
[[package]]
name = "windmill-dep-map"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"chrono",
"itertools 0.14.0",
@@ -16857,7 +16831,7 @@ dependencies = [
[[package]]
name = "windmill-git-sync"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"regex",
"serde",
@@ -16872,7 +16846,7 @@ dependencies = [
[[package]]
name = "windmill-indexer"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -16896,7 +16870,7 @@ dependencies = [
[[package]]
name = "windmill-jseval"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"futures",
@@ -16913,7 +16887,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"itertools 0.14.0",
"lazy_static",
@@ -16929,7 +16903,7 @@ dependencies = [
[[package]]
name = "windmill-mcp"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16950,7 +16924,7 @@ dependencies = [
[[package]]
name = "windmill-native-triggers"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16981,7 +16955,7 @@ dependencies = [
[[package]]
name = "windmill-oauth"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-oauth2",
@@ -17005,7 +16979,7 @@ dependencies = [
[[package]]
name = "windmill-object-store"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-stream",
@@ -17039,7 +17013,7 @@ dependencies = [
[[package]]
name = "windmill-operator"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"futures",
@@ -17057,7 +17031,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"convert_case 0.6.0",
"serde",
@@ -17066,7 +17040,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -17078,7 +17052,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"serde_json",
@@ -17090,7 +17064,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"gosyn",
@@ -17102,7 +17076,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -17114,7 +17088,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"serde_json",
@@ -17126,7 +17100,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -17137,7 +17111,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -17148,7 +17122,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -17160,7 +17134,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -17171,7 +17145,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -17193,7 +17167,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"serde_json",
@@ -17205,7 +17179,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -17219,7 +17193,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"convert_case 0.6.0",
@@ -17236,7 +17210,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -17249,7 +17223,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"serde",
@@ -17261,7 +17235,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -17279,7 +17253,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -17295,7 +17269,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -17311,7 +17285,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"serde",
@@ -17322,7 +17296,7 @@ dependencies = [
[[package]]
name = "windmill-queue"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -17359,7 +17333,7 @@ dependencies = [
[[package]]
name = "windmill-runtime-nativets"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"const_format",
@@ -17397,7 +17371,7 @@ dependencies = [
[[package]]
name = "windmill-sql-datatype-parser-wasm"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"getrandom 0.3.4",
"wasm-bindgen",
@@ -17408,7 +17382,7 @@ dependencies = [
[[package]]
name = "windmill-store"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -17437,7 +17411,7 @@ dependencies = [
[[package]]
name = "windmill-test-utils"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17461,7 +17435,7 @@ dependencies = [
[[package]]
name = "windmill-trigger"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17494,7 +17468,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-email"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17514,7 +17488,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-gcp"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17548,7 +17522,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-http"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17584,7 +17558,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-kafka"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17607,7 +17581,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-mqtt"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17631,7 +17605,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-nats"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-nats",
@@ -17655,7 +17629,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-postgres"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17690,7 +17664,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-sqs"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17718,7 +17692,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-websocket"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17741,7 +17715,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"bitflags 2.9.4",
@@ -17760,7 +17734,7 @@ dependencies = [
[[package]]
name = "windmill-worker"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"anyhow",
"async-once-cell",
@@ -17871,7 +17845,7 @@ dependencies = [
[[package]]
name = "windmill-worker-volumes"
version = "1.679.0"
version = "1.673.0"
dependencies = [
"bytes",
"futures",
@@ -18561,7 +18535,7 @@ dependencies = [
"id-arena",
"indexmap 2.12.0",
"log",
"semver 1.0.28",
"semver 1.0.27",
"serde",
"serde_derive",
"serde_json",

View File

@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.679.0"
version = "1.673.0"
authors.workspace = true
edition.workspace = true
@@ -85,7 +85,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.679.0"
version = "1.673.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -560,7 +560,6 @@ aws-sdk-bedrock = "1.129.0"
aws-sdk-bedrockruntime = "=1.122.0"
aws-credential-types = "^1"
aws-smithy-types = "^1"
aws-sdk-secretsmanager = "^1"
aws-sdk-sqs = "=1.77.0"
aws-sdk-sts = "=1.79.0"
aws-sdk-sso = "=1.77.0"

View File

@@ -1 +1 @@
a30e828a8742b8b644a25a603f0ee380a374607f
ef37ca96f140dcd553226fac6bce3ef6d57ec03d

View File

@@ -1,2 +0,0 @@
DROP INDEX IF EXISTS concurrency_key_ended_at_only_idx;
DROP INDEX IF EXISTS autoscaling_event_applied_at_idx;

View File

@@ -1,7 +0,0 @@
-- Add indexes on the time column alone to speed up the periodic cleanup DELETEs in monitor.rs:
-- DELETE FROM concurrency_key WHERE ended_at <= now() - ($1::bigint::text || ' s')::interval
-- DELETE FROM autoscaling_event WHERE applied_at <= now() - ($1::bigint::text || ' s')::interval
-- The existing composite indexes (key, ended_at DESC) and (worker_group, applied_at) cannot be
-- used efficiently when filtering only on the time column, so Postgres falls back to a seq scan.
CREATE INDEX IF NOT EXISTS concurrency_key_ended_at_only_idx ON concurrency_key (ended_at);
CREATE INDEX IF NOT EXISTS autoscaling_event_applied_at_idx ON autoscaling_event (applied_at);

View File

@@ -1 +0,0 @@
DROP INDEX IF EXISTS idx_resource_cache_expire;

View File

@@ -1,5 +0,0 @@
-- Partial functional index to speed up periodic cleanup of expired cache resources.
-- Matches the query: DELETE FROM resource WHERE resource_type = 'cache' AND to_timestamp((value->>'expire')::int) < now()
CREATE INDEX IF NOT EXISTS idx_resource_cache_expire
ON resource (to_timestamp((value->>'expire')::int))
WHERE resource_type = 'cache';

View File

@@ -1 +0,0 @@
DROP TABLE IF EXISTS background_task_state;

View File

@@ -1,19 +0,0 @@
-- Tracks the live state of long-running background tasks coordinated across
-- multiple server replicas (log cleanup, object-storage usage scan, staggered
-- graceful restart). Previously these were stored in-memory per-process or in
-- `global_settings` (which is conceptually for user-configurable instance
-- settings and is exposed via the settings UI / export).
--
-- `value` holds task-specific JSON. `running`/`owner`/`updated_at` implement
-- a lease: a task is "free" if running=false OR updated_at is older than the
-- heartbeat-stale threshold (handled in application code, see
-- windmill-common/src/background_task.rs).
CREATE TABLE background_task_state (
name TEXT PRIMARY KEY,
value JSONB NOT NULL,
running BOOLEAN NOT NULL DEFAULT FALSE,
owner TEXT,
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

View File

@@ -1 +0,0 @@
ALTER TABLE resource DROP COLUMN IF EXISTS ws_specific;

View File

@@ -1,3 +0,0 @@
-- Add ws_specific flag to resource table.
-- When true, the resource is excluded from workspace diff comparisons (e.g. auto-created resources during fork).
ALTER TABLE resource ADD COLUMN IF NOT EXISTS ws_specific BOOLEAN NOT NULL DEFAULT false;

View File

@@ -1,2 +0,0 @@
DROP TABLE IF EXISTS job_delete_schedule;
ALTER TABLE script DROP COLUMN IF EXISTS delete_after_secs;

View File

@@ -1,9 +0,0 @@
ALTER TABLE script ADD COLUMN delete_after_secs INTEGER;
CREATE TABLE job_delete_schedule (
job_id UUID PRIMARY KEY,
workspace_id VARCHAR(50) NOT NULL,
delete_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_job_delete_schedule_delete_at ON job_delete_schedule (delete_at);

View File

@@ -106,28 +106,6 @@ fn extract_wmill_datatable_call(expr: &Expr) -> Option<(AssetKind, String, Optio
}
None
}
/// Check if an expression is `tag.raw(...)` where `tag` matches the given tag name.
/// Returns true for patterns like `sql.raw(x)`.
fn is_raw_call(expr: &Expr, tag_name: &str) -> bool {
if let Expr::Call(call_expr) = expr {
if let Some(Expr::Member(member)) = call_expr.callee.as_expr().map(AsRef::as_ref) {
let is_tag = matches!(
member.obj.as_ref(),
Expr::Ident(ident) if ident.sym.as_str() == tag_name
);
if is_tag {
if let MemberProp::Ident(prop) = &member.prop {
return prop.sym.as_str() == "raw";
}
}
}
}
false
}
const WM_SQL_RAW_PLACEHOLDER: &str = "__WM_SQL_RAW__";
impl Visit for AssetsFinder {
// visit_call_expr will not recurse if it detects an asset,
// so this will only be called when no further context was found
@@ -252,31 +230,21 @@ impl Visit for AssetsFinder {
return;
};
// Determine which interpolations are sql.raw() calls
let raw_flags: Vec<bool> = node
.tpl
.exprs
.iter()
.map(|expr| is_raw_call(expr.as_ref(), tag_name))
.collect();
let has_raw_interpolation = raw_flags.iter().any(|&r| r);
// Extract the SQL query from the template quasis (string parts)
// Substitute ${} with $N for normal args, __WM_SQL_RAW__ for raw args
let mut sql = String::new();
let mut arg_index = 0usize;
for (i, quasi) in node.tpl.quasis.iter().enumerate() {
if i > 0 {
let is_raw = raw_flags.get(i - 1).copied().unwrap_or(false);
if is_raw {
sql.push_str(WM_SQL_RAW_PLACEHOLDER);
// Substitute ${} with $1, $2, etc.
let sql: String = node
.tpl
.quasis
.iter()
.map(|quasi| quasi.raw.as_str())
.enumerate()
.fold(String::new(), |acc, (i, s)| {
if i == 0 {
s.to_string()
} else {
arg_index += 1;
sql.push_str(&format!("${}", arg_index));
format!("{}${}{}", acc, i, s)
}
}
sql.push_str(quasi.raw.as_str());
}
});
// Capture SQL query details before transforming for SQL parser
let span = node.span();
@@ -288,7 +256,6 @@ impl Visit for AssetsFinder {
source_kind: *kind,
source_name: asset_name.clone(),
source_schema: schema.clone(),
has_raw_interpolation,
});
// We use the SQL parser to detect RW, specific tables, etc.
@@ -299,11 +266,7 @@ impl Visit for AssetsFinder {
&sql,
);
match sql_assets {
Ok(Some(sql_assets)) => self.assets.extend(
sql_assets
.into_iter()
.filter(|a| !a.path.contains(WM_SQL_RAW_PLACEHOLDER)),
),
Ok(Some(sql_assets)) => self.assets.extend(sql_assets),
_ => {}
}
}
@@ -666,7 +629,6 @@ mod tests {
assert_eq!(query_detail.source_kind, AssetKind::DataTable);
assert_eq!(query_detail.source_name, "dt");
assert_eq!(query_detail.source_schema, None);
assert_eq!(query_detail.has_raw_interpolation, false);
// Span should be non-zero
assert!(query_detail.span.0 > 0);
assert!(query_detail.span.1 > query_detail.span.0);
@@ -730,86 +692,5 @@ mod tests {
assert_eq!(query_detail.source_kind, AssetKind::Ducklake);
assert_eq!(query_detail.source_name, "my_lake");
assert_eq!(query_detail.source_schema, None);
assert_eq!(query_detail.has_raw_interpolation, false);
}
#[test]
fn test_ts_asset_parser_sql_raw_basic() {
let input = r#"
import * as wmill from "windmill-client"
export async function main(table: string) {
let sql = wmill.datatable('dt')
return await sql`SELECT * FROM ${sql.raw(table)}`.fetch()
}
"#;
let result = parse_assets(input).unwrap();
// sql.raw in table position => the __WM_SQL_RAW__ asset gets filtered out,
// but the datatable itself is still tracked as "used" (without specific table info)
assert_eq!(
result.assets,
vec![ParseAssetsResult {
kind: AssetKind::DataTable,
path: "dt".to_string(),
access_type: None,
columns: None,
}]
);
// Check SQL query details
assert_eq!(result.sql_queries.len(), 1);
let q = &result.sql_queries[0];
assert!(q.query_string.contains("__WM_SQL_RAW__"));
assert!(!q.query_string.contains("$1"));
assert_eq!(q.has_raw_interpolation, true);
assert_eq!(q.source_kind, AssetKind::DataTable);
assert_eq!(q.source_name, "dt");
}
#[test]
fn test_ts_asset_parser_sql_raw_mixed() {
let input = r#"
import * as wmill from "windmill-client"
export async function main(name: string, col: string, val: number) {
let sql = wmill.datatable('dt')
return await sql`SELECT * FROM users WHERE name = ${name} AND ${sql.raw(col)} = ${val}`.fetch()
}
"#;
let result = parse_assets(input).unwrap();
// "users" table should still be detected
assert_eq!(result.assets.len(), 1);
assert_eq!(result.assets[0].path, "dt/users");
assert_eq!(result.assets[0].access_type, Some(R));
// Check SQL query details — arg numbering skips the raw interpolation
assert_eq!(result.sql_queries.len(), 1);
let q = &result.sql_queries[0];
assert_eq!(
q.query_string,
"SELECT * FROM users WHERE name = $1 AND __WM_SQL_RAW__ = $2"
);
assert_eq!(q.has_raw_interpolation, true);
}
#[test]
fn test_ts_asset_parser_sql_raw_no_false_positive() {
// Ensure that a normal query (no sql.raw) still has has_raw_interpolation=false
let input = r#"
import * as wmill from "windmill-client"
export async function main(x: number) {
let sql = wmill.datatable('dt')
return await sql`SELECT * FROM friends WHERE age = ${x}`.fetch()
}
"#;
let result = parse_assets(input).unwrap();
assert_eq!(result.sql_queries.len(), 1);
assert_eq!(result.sql_queries[0].has_raw_interpolation, false);
assert_eq!(
result.sql_queries[0].query_string,
"SELECT * FROM friends WHERE age = $1"
);
assert_eq!(result.assets.len(), 1);
assert_eq!(result.assets[0].path, "dt/friends");
}
}

View File

@@ -39,8 +39,6 @@ pub struct SqlQueryDetails {
pub source_name: String, // e.g., "main", "dt"
#[serde(skip_serializing_if = "Option::is_none")]
pub source_schema: Option<String>, // e.g., Some("public"), None
#[serde(skip_serializing_if = "std::ops::Not::not")]
pub has_raw_interpolation: bool, // true if any ${sql.raw(...)} was used
}
#[derive(Serialize, Debug, Default)]

View File

@@ -754,7 +754,7 @@ async fn windmill_main() -> anyhow::Result<()> {
.and_then(|x| x.parse().ok())
.unwrap_or(IpAddr::from(default_bind_addr));
let (conn, first_suffix, agent_config) = if matches!(mode, Mode::Agent | Mode::AgentBatch) {
let (conn, first_suffix, agent_config) = if mode == Mode::Agent {
let agent_config = match AgentConfig::from_env() {
Ok(config) => config,
Err(e) => {
@@ -829,7 +829,7 @@ async fn windmill_main() -> anyhow::Result<()> {
let _guard = windmill_common::tracing_init::initialize_tracing(&hostname, &mode, &environment);
let is_agent = matches!(mode, Mode::Agent | Mode::AgentBatch);
let is_agent = mode == Mode::Agent;
let mut migration_handle: Option<JoinHandle<()>> = None;
#[cfg(feature = "parquet")]
@@ -953,7 +953,7 @@ async fn windmill_main() -> anyhow::Result<()> {
}
}
let conn = if matches!(mode, Mode::Agent | Mode::AgentBatch) {
let conn = if mode == Mode::Agent {
conn
} else {
// Drop the initial connection pool before creating the main one.
@@ -1289,7 +1289,7 @@ Windmill Community Edition {GIT_VERSION}
)
},
worker_name: worker_name_with_suffix(
matches!(mode, Mode::Agent | Mode::AgentBatch),
mode == Mode::Agent,
WORKER_GROUP.as_str(),
&suffix,
),

View File

@@ -1205,92 +1205,6 @@ pub async fn delete_expired_items(db: &DB) -> () {
}
}
#[cfg(feature = "enterprise")]
async fn cleanup_scheduled_job_deletions(db: &Pool<Postgres>) {
const BATCH_SIZE: i64 = 1000;
const MAX_BATCHES: i32 = 10;
let mut total_deleted = 0u64;
for batch_num in 0..MAX_BATCHES {
let mut tx = match db.begin().await {
Ok(tx) => tx,
Err(e) => {
tracing::error!("Error starting transaction for scheduled job deletion: {e:?}");
break;
}
};
let rows = match sqlx::query_scalar!(
"DELETE FROM job_delete_schedule
WHERE job_id IN (
SELECT job_id FROM job_delete_schedule
WHERE delete_at <= now()
ORDER BY delete_at
LIMIT $1
FOR UPDATE SKIP LOCKED
)
RETURNING job_id",
BATCH_SIZE,
)
.fetch_all(&mut *tx)
.await
{
Ok(rows) => rows,
Err(e) => {
tracing::error!("Error in scheduled job deletion batch {batch_num}: {e:?}");
break;
}
};
if rows.is_empty() {
break;
}
let job_ids = rows;
let count = job_ids.len() as u64;
let cleanup_result: Result<(), sqlx::Error> = async {
sqlx::query!(
"UPDATE v2_job SET args = '{}'::jsonb WHERE id = ANY($1)",
&job_ids,
)
.execute(&mut *tx)
.await?;
sqlx::query!(
"UPDATE v2_job_completed SET result = '{}'::jsonb WHERE id = ANY($1)",
&job_ids,
)
.execute(&mut *tx)
.await?;
sqlx::query!(
"UPDATE job_logs SET logs = '##DELETED##' WHERE job_id = ANY($1)",
&job_ids,
)
.execute(&mut *tx)
.await?;
Ok(())
}
.await;
if let Err(e) = cleanup_result {
// Roll back so schedule rows survive for retry on next cycle
tracing::error!("Error cleaning job data in batch {batch_num}, rolling back: {e:?}");
break;
}
if let Err(e) = tx.commit().await {
tracing::error!("Error committing scheduled job deletion batch {batch_num}: {e:?}");
break;
}
total_deleted += count;
}
if total_deleted > 0 {
tracing::info!("Scheduled job deletion: cleaned {total_deleted} jobs");
}
}
pub async fn check_expiring_tokens(db: &DB) {
// Find tokens expiring within 7 days that still have a pending notification row.
// The notification table stores token_hash (not plaintext) so the join works
@@ -1438,37 +1352,18 @@ async fn delete_log_files_from_disk_and_store(
tmp_dir: &str,
_s3_prefix: &str,
) {
// S3 bulk delete (batched via delete_stream — on S3 this uses the DeleteObjects
// API, up to 1000 objects per request).
#[cfg(feature = "parquet")]
{
let should_del_from_store = *MONITOR_LOGS_ON_OBJECT_STORE.read().await;
if should_del_from_store {
if let Some(os) = windmill_object_store::get_object_store().await {
let s3_paths: Vec<_> = paths_to_delete
.iter()
.map(|p| {
windmill_object_store::object_store_reexports::Path::from(format!(
"{}{}",
_s3_prefix, p
))
})
.map(Ok)
.collect();
let stream = futures::stream::iter(s3_paths).boxed();
let mut result = os.delete_stream(stream);
while let Some(r) = result.next().await {
if let Err(e) = r {
tracing::error!("Failed to delete from object store: {e}");
}
}
}
}
}
let os = windmill_object_store::get_object_store().await;
#[cfg(not(feature = "parquet"))]
let os: Option<()> = None;
let _should_del_from_store = MONITOR_LOGS_ON_OBJECT_STORE.read().await.clone();
// Disk delete in parallel.
let delete_futures = FuturesUnordered::new();
for path in paths_to_delete {
let _os2 = &os;
delete_futures.push(async move {
let disk_path = std::path::Path::new(tmp_dir).join(&path);
if tokio::fs::metadata(&disk_path).await.is_ok() {
@@ -1477,10 +1372,31 @@ async fn delete_log_files_from_disk_and_store(
"Failed to delete from disk {}: {e}",
disk_path.to_string_lossy()
);
} else {
tracing::debug!(
"Succesfully deleted {} from disk",
disk_path.to_string_lossy()
);
}
}
#[cfg(feature = "parquet")]
if _should_del_from_store {
if let Some(os) = _os2 {
let p = windmill_object_store::object_store_reexports::Path::from(format!(
"{}{}",
_s3_prefix, path
));
if let Err(e) = os.delete(&p).await {
tracing::error!("Failed to delete from object store {}: {e}", p.to_string())
} else {
tracing::debug!("Succesfully deleted {} from object store", p.to_string());
}
}
}
});
}
let _: Vec<_> = delete_futures.collect().await;
}
@@ -2414,15 +2330,6 @@ pub async fn monitor_db(
}
};
let cleanup_scheduled_job_deletions_f = async {
#[cfg(feature = "enterprise")]
if server_mode && !initial_load {
if let Some(db) = conn.as_sql() {
cleanup_scheduled_job_deletions(&db).await;
}
}
};
join!(
expired_items_f,
zombie_jobs_f,
@@ -2446,7 +2353,6 @@ pub async fn monitor_db(
cleanup_notify_events_f,
check_expiring_tokens_f,
manage_audit_partitions_f,
cleanup_scheduled_job_deletions_f,
);
}

View File

@@ -1,92 +0,0 @@
use sqlx::{Pool, Postgres};
use windmill_common::jobs::{resolve_delete_after_secs, schedule_job_deletion};
use windmill_test_utils::*;
// ---------------------------------------------------------------------------
// Unit tests for resolve_delete_after_secs
// ---------------------------------------------------------------------------
#[test]
fn test_resolve_no_deletion() {
assert_eq!(resolve_delete_after_secs(None, None), None);
assert_eq!(resolve_delete_after_secs(Some(false), None), None);
}
#[test]
fn test_resolve_immediate_backward_compat() {
// delete_after_use=true with no secs → immediate (0)
assert_eq!(resolve_delete_after_secs(Some(true), None), Some(0));
}
#[test]
fn test_resolve_explicit_secs() {
assert_eq!(resolve_delete_after_secs(None, Some(0)), Some(0));
assert_eq!(resolve_delete_after_secs(None, Some(3600)), Some(3600));
assert_eq!(resolve_delete_after_secs(Some(true), Some(60)), Some(60));
assert_eq!(resolve_delete_after_secs(Some(false), Some(120)), Some(120));
}
#[test]
fn test_resolve_rejects_negative() {
assert_eq!(resolve_delete_after_secs(None, Some(-1)), None);
assert_eq!(resolve_delete_after_secs(Some(true), Some(-100)), None);
}
// ---------------------------------------------------------------------------
// Integration: schedule_job_deletion inserts into job_delete_schedule
// ---------------------------------------------------------------------------
#[sqlx::test(fixtures("base"))]
async fn test_schedule_job_deletion_inserts_row(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let job_id = uuid::Uuid::new_v4();
schedule_job_deletion(&db, job_id, "test-workspace", 3600).await?;
let row = sqlx::query_as::<_, (uuid::Uuid, String)>(
"SELECT job_id, workspace_id FROM job_delete_schedule WHERE job_id = $1",
)
.bind(job_id)
.fetch_one(&db)
.await?;
assert_eq!(row.0, job_id);
assert_eq!(row.1, "test-workspace");
// Verify delete_at is approximately now + 3600s
let delete_at: chrono::DateTime<chrono::Utc> =
sqlx::query_scalar("SELECT delete_at FROM job_delete_schedule WHERE job_id = $1")
.bind(job_id)
.fetch_one(&db)
.await?;
let expected_min = chrono::Utc::now() + chrono::Duration::seconds(3500);
let expected_max = chrono::Utc::now() + chrono::Duration::seconds(3700);
assert!(
delete_at > expected_min && delete_at < expected_max,
"delete_at should be ~1 hour from now, got {delete_at}"
);
Ok(())
}
#[sqlx::test(fixtures("base"))]
async fn test_schedule_job_deletion_is_idempotent(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let job_id = uuid::Uuid::new_v4();
schedule_job_deletion(&db, job_id, "test-workspace", 60).await?;
// Second call should not error (ON CONFLICT DO NOTHING)
schedule_job_deletion(&db, job_id, "test-workspace", 120).await?;
let count: i64 =
sqlx::query_scalar("SELECT COUNT(*) FROM job_delete_schedule WHERE job_id = $1")
.bind(job_id)
.fetch_one(&db)
.await?;
assert_eq!(count, 1, "should have exactly one row (idempotent)");
Ok(())
}

View File

@@ -35,7 +35,7 @@ mod dependency_map {
schema: std::collections::HashMap::new(),
ws_error_handler_muted: Some(false),
priority: None,
delete_after_secs: None,
delete_after_use: None,
timeout: None,
restart_unless_cancelled: None,
deployment_message: None,

File diff suppressed because it is too large Load Diff

View File

@@ -395,14 +395,6 @@ async fn test_root_job_span_created_on_success() {
attrs.contains(&"script_path"),
"missing script_path attribute"
);
assert!(
attrs.contains(&"job_kind"),
"missing job_kind attribute"
);
assert!(
attrs.contains(&"created_by"),
"missing created_by attribute"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]

View File

@@ -33,7 +33,7 @@ mod relock_skip {
schema: std::collections::HashMap::new(),
ws_error_handler_muted: Some(false),
priority: None,
delete_after_secs: None,
delete_after_use: None,
timeout: None,
restart_unless_cancelled: None,
deployment_message: None,

View File

@@ -1,5 +1,4 @@
use serde::de::DeserializeOwned;
use serial_test::serial;
#[cfg(feature = "enterprise")]
use chrono::Timelike;
@@ -206,7 +205,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -254,7 +252,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -276,7 +273,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -392,7 +388,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -449,7 +444,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -492,7 +486,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -514,7 +507,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -563,7 +555,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -1514,10 +1505,8 @@ export async function main(a: Date) {
}
#[sqlx::test(fixtures("base"))]
#[serial(pg_cache)]
async fn test_postgresql_job(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
windmill_worker::pg_executor::clear_pg_cache().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
@@ -1555,311 +1544,6 @@ SELECT 'hello ' || $1::text AS result;
Ok(())
}
#[sqlx::test(fixtures("base"))]
#[serial(pg_cache)]
async fn test_postgresql_cached_connection_resets_session(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
use std::sync::atomic::Ordering;
use windmill_worker::pg_executor::{clear_pg_cache, CACHE_HITS};
initialize_tracing().await;
// Clear stale connections from prior tests.
clear_pg_cache().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let hits_before = CACHE_HITS.load(Ordering::Relaxed);
let db_arg = json!({"host": "localhost", "port": 5432, "dbname": "windmill", "user": "postgres", "password": "changeme"});
let make_pg_job = |content: String| {
RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
language: ScriptLang::Postgresql,
cache_ttl: None,
cache_ignore_s3_path: None,
dedicated_worker: None,
concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default(
)
.into(),
debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(),
modules: None,
}))
.arg("database", db_arg.clone())
};
// Run 3 jobs against the same database resource so that at least the
// second→third transition exercises the cached-connection path.
//
// Job 1: warm up — creates and (if eligible) caches the connection.
let result1 = make_pg_job("SELECT 1 as n;".into())
.run_until_complete(&db, false, port)
.await
.json_result()
.unwrap();
assert_eq!(result1, json!([{"n": 1}]));
// Job 2: mutate session state — set a custom search_path.
let result2 = make_pg_job(
"SET search_path TO pg_catalog;\nSELECT current_setting('search_path') as sp;".into(),
)
.run_until_complete(&db, false, port)
.await
.json_result()
.unwrap();
assert_eq!(
result2,
json!([{"sp": "pg_catalog"}]),
"second job should see the custom search_path it just set"
);
// Job 3: read search_path — RESET ALL (on cached conn) should have restored
// the default.
let result3 = make_pg_job("SELECT current_setting('search_path') as sp;".into())
.run_until_complete(&db, false, port)
.await
.json_result()
.unwrap();
let sp = result3[0]["sp"].as_str().unwrap();
assert_ne!(
sp, "pg_catalog",
"search_path must be reset between jobs, got: {sp}"
);
// Verify the cached-connection path was actually exercised.
let hits_after = CACHE_HITS.load(Ordering::Relaxed);
assert!(
hits_after > hits_before,
"expected at least one cache hit across the 3 jobs, got {hits_before} -> {hits_after}"
);
Ok(())
}
/// Runs multiple PG jobs through a SINGLE worker (like production) to verify
/// that SET ROLE / search_path changes do not leak across jobs.
#[sqlx::test(fixtures("base"))]
#[serial(pg_cache)]
async fn test_postgresql_single_worker_session_isolation(db: Pool<Postgres>) -> anyhow::Result<()> {
use std::sync::atomic::Ordering;
use windmill_common::worker::Connection;
use windmill_worker::pg_executor::{clear_pg_cache, CACHE_HITS};
initialize_tracing().await;
clear_pg_cache().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let hits_before = CACHE_HITS.load(Ordering::Relaxed);
let db_arg = json!({"host": "localhost", "port": 5432, "dbname": "windmill", "user": "postgres", "password": "changeme"});
let make_pg_job = |content: String| {
RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
language: ScriptLang::Postgresql,
cache_ttl: None,
cache_ignore_s3_path: None,
dedicated_worker: None,
concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default(
)
.into(),
debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(),
modules: None,
}))
.arg("database", db_arg.clone())
};
// Push all 3 jobs into the queue BEFORE starting the worker, so a single
// worker processes them all sequentially — matching production behaviour.
let id1 = make_pg_job("SELECT current_user as cu;".into())
.push(&db)
.await;
let id2 = make_pg_job("SET ROLE postgres;\nSET search_path TO pg_catalog;\nSELECT current_user as cu, current_setting('search_path') as sp;".into())
.push(&db)
.await;
let id3 =
make_pg_job("SELECT current_user as cu, current_setting('search_path') as sp;".into())
.push(&db)
.await;
// Run ONE worker that processes all three jobs.
let ids = [id1, id2, id3];
let listener = listen_for_completed_jobs(&db).await;
in_test_worker(
Connection::Sql(db.clone()),
async {
use futures::StreamExt;
let mut remaining = ids
.iter()
.copied()
.collect::<std::collections::HashSet<_>>();
let mut listener = listener;
while !remaining.is_empty() {
if let Some(id) = listener.next().await {
remaining.remove(&id);
}
}
},
port,
)
.await;
let r1 = completed_job(id1, &db).await.json_result().unwrap();
let r2 = completed_job(id2, &db).await.json_result().unwrap();
let r3 = completed_job(id3, &db).await.json_result().unwrap();
// Job 1: baseline — should be the connecting user (postgres)
assert_eq!(r1[0]["cu"], "postgres");
// Job 2: SET ROLE + SET search_path took effect within the job
assert_eq!(r2[0]["sp"], "pg_catalog");
// Job 3: neither the role nor the search_path should leak from job 2
let cu3 = r3[0]["cu"].as_str().unwrap();
let sp3 = r3[0]["sp"].as_str().unwrap();
assert_eq!(
cu3, "postgres",
"SET ROLE must not leak across jobs, got: {cu3}"
);
assert_ne!(
sp3, "pg_catalog",
"search_path must not leak across jobs, got: {sp3}"
);
// Verify caching actually happened (jobs 2 and 3 should hit the cache).
let hits_after = CACHE_HITS.load(Ordering::Relaxed);
assert!(
hits_after > hits_before,
"expected cache hits, got {hits_before} -> {hits_after}"
);
Ok(())
}
/// Runs 100 varied PG jobs through a single worker, verifying every job
/// succeeds, the cache is used for 99 of them, and session state never leaks.
#[sqlx::test(fixtures("base"))]
#[serial(pg_cache)]
async fn test_postgresql_100_jobs_cached(db: Pool<Postgres>) -> anyhow::Result<()> {
use std::sync::atomic::Ordering;
use windmill_common::worker::Connection;
use windmill_worker::pg_executor::{clear_pg_cache, CACHE_HITS};
initialize_tracing().await;
clear_pg_cache().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let hits_before = CACHE_HITS.load(Ordering::Relaxed);
let db_arg = json!({"host": "localhost", "port": 5432, "dbname": "windmill", "user": "postgres", "password": "changeme"});
let make_pg_job = |content: String| {
RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
language: ScriptLang::Postgresql,
cache_ttl: None,
cache_ignore_s3_path: None,
dedicated_worker: None,
concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default(
)
.into(),
debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(),
modules: None,
}))
.arg("database", db_arg.clone())
};
const N: usize = 100;
let mut ids = Vec::with_capacity(N);
// Push 100 varied jobs — mix of session-mutating and read-only queries.
for i in 0..N {
let content = match i % 5 {
// Plain SELECT with different values each time
0 => format!("SELECT {} as n, current_user as cu, current_setting('search_path') as sp;", i),
// SET search_path then read it
1 => format!("SET search_path TO pg_catalog;\nSELECT {} as n, current_setting('search_path') as sp;", i),
// SET ROLE then read it
2 => "SET ROLE postgres;\nSELECT current_user as cu;".to_string(),
// Multi-statement
3 => format!("SELECT 1;\nSELECT {} as n;", i),
// Read-only with math
_ => format!("SELECT {} + {} as n;", i, i * 2),
};
ids.push(make_pg_job(content).push(&db).await);
}
// Run ONE worker for all 100 jobs.
let listener = listen_for_completed_jobs(&db).await;
let id_set = ids
.iter()
.copied()
.collect::<std::collections::HashSet<_>>();
in_test_worker(
Connection::Sql(db.clone()),
async {
use futures::StreamExt;
let mut remaining = id_set;
let mut listener = listener;
while !remaining.is_empty() {
if let Some(id) = listener.next().await {
remaining.remove(&id);
}
}
},
port,
)
.await;
// Verify all 100 jobs succeeded.
for (i, id) in ids.iter().enumerate() {
let cjob = completed_job(*id, &db).await;
assert!(cjob.success, "job {i} (id={id}) failed: {:?}", cjob.result);
}
// Spot-check: jobs that followed a SET search_path should NOT see pg_catalog.
// Pattern: job i%5==1 sets search_path, job i%5==2 follows — should be clean.
for i in (2..N).step_by(5) {
let cjob = completed_job(ids[i], &db).await;
let result = cjob.json_result().unwrap();
let cu = result[0]["cu"].as_str().unwrap_or("N/A");
assert_eq!(cu, "postgres", "job {i}: SET ROLE leaked, got {cu}");
}
for i in (0..N).step_by(5) {
let cjob = completed_job(ids[i], &db).await;
let result = cjob.json_result().unwrap();
let sp = result[0]["sp"].as_str().unwrap_or("N/A");
assert_ne!(sp, "pg_catalog", "job {i}: search_path leaked, got {sp}");
}
// Verify caching: first job creates the connection, remaining 99 should hit cache.
let hits_after = CACHE_HITS.load(Ordering::Relaxed);
let new_hits = hits_after - hits_before;
assert!(
new_hits >= (N as u64 - 1),
"expected at least {} cache hits, got {new_hits}",
N - 1
);
Ok(())
}
#[cfg(feature = "mysql")]
#[sqlx::test(fixtures("base"))]
async fn test_mysql_job(db: Pool<Postgres>) -> anyhow::Result<()> {

View File

@@ -608,12 +608,11 @@ fn extract_domain_from_route(
)))
}
const RUN_WHITELISTED_GET_PATHS: [&'static str; 20] = [
const RUN_WHITELISTED_GET_PATHS: [&'static str; 19] = [
"jobs_u/get_flow/",
"jobs_u/get_root_job_id/",
"jobs_u/get/",
"jobs_u/get_logs/",
"jobs_u/get_flow_all_logs/",
"jobs_u/get_args/",
"jobs_u/get_flow_debug_info/",
"jobs_u/completed/get/",

View File

@@ -364,7 +364,7 @@ pub mod types {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub dedicated_worker: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delete_after_secs: Option<i32>,
pub delete_after_use: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub deployment_message: Option<String>,
pub description: String,
@@ -540,7 +540,7 @@ pub mod types {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub continue_on_error: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delete_after_secs: Option<i32>,
pub delete_after_use: Option<bool>,
pub id: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub mock: Option<serde_json::Value>,

View File

@@ -1856,7 +1856,6 @@ mod tests {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -1891,7 +1890,6 @@ mod tests {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -1926,7 +1924,6 @@ mod tests {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -1960,7 +1957,6 @@ mod tests {
timeout: None,
priority: None,
delete_after_use: None,
delete_after_secs: None,
continue_on_error: None,
skip_if: None,
apply_preprocessor: None,
@@ -1976,8 +1972,6 @@ mod tests {
early_return: None,
chat_input_enabled: None,
flow_env: None,
delete_after_use: None,
delete_after_secs: None,
concurrency_settings: ConcurrencySettings::default(),
debouncing_settings: DebouncingSettings::default(),
};

View File

@@ -37,8 +37,6 @@ async fn start_mock_ai_api() -> u16 {
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_ai_proxy_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
// Allow the mock AI API on 127.0.0.1 to pass the SSRF check
std::env::set_var("ALLOW_PRIVATE_AI_BASE_URLS", "true");
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();

View File

@@ -159,7 +159,12 @@ async fn test_app_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "public_app: {}", resp.text().await?);
assert_eq!(
resp.status(),
200,
"public_app: {}",
resp.text().await?
);
// --- secret_of_latest_version ---
let resp = authed_get(port, "secret_of_latest_version", "u/test-user/test_app").await;
@@ -181,6 +186,7 @@ async fn test_app_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
resp.text().await?
);
// --- history_update ---
let app_body = authed_get(port, "get/p", "u/test-user/test_app").await;
let app = app_body.json::<serde_json::Value>().await?;
@@ -192,7 +198,12 @@ async fn test_app_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "history_update: {}", resp.text().await?);
assert_eq!(
resp.status(),
200,
"history_update: {}",
resp.text().await?
);
// --- update ---
let resp = authed(client().post(app_url(port, "update", "u/test-user/test_app")))
@@ -228,10 +239,12 @@ async fn test_app_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
// ===== Hub endpoints (require external network, expect 500 or 200) =====
// --- hub/list ---
let resp = authed(client().get(format!("http://localhost:{port}/api/apps/hub/list")))
.send()
.await
.unwrap();
let resp = authed(client().get(format!(
"http://localhost:{port}/api/apps/hub/list"
)))
.send()
.await
.unwrap();
assert!(
resp.status() == 200 || resp.status() == 500,
"hub/list: unexpected status {}",
@@ -239,10 +252,12 @@ async fn test_app_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
);
// --- hub/get ---
let resp = authed(client().get(format!("http://localhost:{port}/api/apps/hub/get/1")))
.send()
.await
.unwrap();
let resp = authed(client().get(format!(
"http://localhost:{port}/api/apps/hub/get/1"
)))
.send()
.await
.unwrap();
assert!(
resp.status() == 200 || resp.status() == 500,
"hub/get: unexpected status {}",
@@ -250,10 +265,12 @@ async fn test_app_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
);
// --- hub/get_raw ---
let resp = authed(client().get(format!("http://localhost:{port}/api/apps/hub/get_raw/1")))
.send()
.await
.unwrap();
let resp = authed(client().get(format!(
"http://localhost:{port}/api/apps/hub/get_raw/1"
)))
.send()
.await
.unwrap();
assert!(
resp.status() == 200 || resp.status() == 500,
"hub/get_raw: unexpected status {}",
@@ -262,82 +279,3 @@ async fn test_app_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
Ok(())
}
#[cfg(feature = "enterprise")]
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_public_app_by_custom_path(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let base = format!("http://localhost:{port}/api/w/test-workspace/apps");
// create app with anonymous execution mode
let resp = authed(client().post(format!("{base}/create")))
.json(&new_app("u/test-user/custom_path_app", "Custom path app"))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 201, "create: {}", resp.text().await?);
// set custom_path on the app
let resp = authed(client().post(app_url(port, "update", "u/test-user/custom_path_app")))
.json(&serde_json::json!({
"custom_path": "my-custom-app",
"policy": {
"execution_mode": "anonymous",
"triggerables": {},
"on_behalf_of": null,
"on_behalf_of_email": null
}
}))
.send()
.await
.unwrap();
assert_eq!(
resp.status(),
200,
"update custom_path: {}",
resp.text().await?
);
// fetch via public_app_by_custom_path (no workspace prefix: CLOUD_HOSTED=false, APP_WORKSPACED_ROUTE=false)
let resp = client()
.get(format!(
"http://localhost:{port}/api/apps_u/public_app_by_custom_path/my-custom-app"
))
.send()
.await
.unwrap();
assert_eq!(
resp.status(),
200,
"public_app_by_custom_path: {}",
resp.text().await?
);
// verify response contains expected fields
let resp = client()
.get(format!(
"http://localhost:{port}/api/apps_u/public_app_by_custom_path/my-custom-app"
))
.send()
.await
.unwrap();
let body = resp.json::<serde_json::Value>().await?;
assert_eq!(body["path"], "u/test-user/custom_path_app");
assert_eq!(body["summary"], "Custom path app");
assert_eq!(body["workspace_id"], "test-workspace");
assert_eq!(body["custom_path"], "my-custom-app");
// nonexistent custom path returns 404
let resp = client()
.get(format!(
"http://localhost:{port}/api/apps_u/public_app_by_custom_path/nonexistent"
))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 404);
Ok(())
}

View File

@@ -476,25 +476,6 @@ pub async fn delete_job_metadata_after_use(db: &DB, job_uuid: Uuid) -> Result<()
Ok(())
}
pub use windmill_common::jobs::resolve_delete_after_secs;
pub use windmill_common::jobs::schedule_job_deletion;
/// Handle deletion or scheduling for a completed job.
pub async fn handle_delete_after_completion(
db: &DB,
job_uuid: Uuid,
w_id: &str,
delete_after_use: Option<bool>,
delete_after_secs: Option<i32>,
) -> Result<(), Error> {
match resolve_delete_after_secs(delete_after_use, delete_after_secs) {
Some(0) => delete_job_metadata_after_use(db, job_uuid).await?,
Some(secs) => schedule_job_deletion(db, job_uuid, w_id, secs).await?,
None => {}
}
Ok(())
}
// ---------------------------------------------------------------------------
// Queue limit check
// ---------------------------------------------------------------------------
@@ -818,7 +799,7 @@ pub async fn push_script_job_by_path_into_queue<'c>(
trigger: Option<TriggerMetadata>,
) -> error::Result<(
Uuid,
Option<i32>,
Option<bool>,
Option<sqlx::Transaction<'c, sqlx::Postgres>>,
)> {
#[cfg(feature = "enterprise")]
@@ -828,16 +809,14 @@ pub async fn push_script_job_by_path_into_queue<'c>(
check_scopes(&authed, || format!("jobs:run:scripts:{script_path}"))?;
let userdb_authed = UserDbWithAuthed { db: user_db.clone(), authed: &authed.to_authed_ref() };
let (job_payload, tag, delete_after_use, delete_after_secs, timeout, on_behalf_of) =
script_path_to_payload(
script_path,
Some(userdb_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?;
let resolved_delete_secs = resolve_delete_after_secs(delete_after_use, delete_after_secs);
let (job_payload, tag, delete_after_use, timeout, on_behalf_of) = script_path_to_payload(
script_path,
Some(userdb_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?;
let scheduled_for = run_query.get_scheduled_for(&db).await?;
let tag = run_query.tag.clone().or(tag);
@@ -907,9 +886,9 @@ pub async fn push_script_job_by_path_into_queue<'c>(
// If we were given a transaction, return it; otherwise commit it
if return_tx {
Ok((uuid, resolved_delete_secs, Some(tx)))
Ok((uuid, delete_after_use, Some(tx)))
} else {
tx.commit().await?;
Ok((uuid, resolved_delete_secs, None))
Ok((uuid, delete_after_use, None))
}
}

View File

@@ -113,8 +113,6 @@ pub struct ScriptWDraft<SR> {
#[serde(skip_serializing_if = "Option::is_none")]
pub delete_after_use: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub delete_after_secs: Option<i32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub timeout: Option<i32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub visible_to_runner_only: Option<bool>,
@@ -180,7 +178,6 @@ impl ScriptWDraft<ScriptRunnableSettingsHandle> {
priority: self.priority,
restart_unless_cancelled: self.restart_unless_cancelled,
delete_after_use: self.delete_after_use,
delete_after_secs: self.delete_after_secs,
timeout: self.timeout,
visible_to_runner_only: self.visible_to_runner_only,
auto_kind: self.auto_kind,
@@ -946,8 +943,8 @@ async fn create_script_internal<'c>(
content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, \
draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, \
dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, \
delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels) \
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41)",
delete_after_use, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels) \
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40)",
&w_id,
&hash.0,
ns.path,
@@ -973,13 +970,12 @@ async fn create_script_internal<'c>(
ns.priority,
ns.restart_unless_cancelled,
ns.delete_after_use,
ns.delete_after_secs,
ns.timeout,
guarded_concurrency_key,
ns.visible_to_runner_only,
auto_kind.as_deref(),
codebase,
has_preprocessor.filter(|x: &bool| *x),
has_preprocessor.filter(|x: &bool| *x), // should be Some(true) or None
windmill_common::resolve_on_behalf_of_email(
ns.on_behalf_of_email.as_deref(),
ns.preserve_on_behalf_of.unwrap_or(false),
@@ -1442,7 +1438,7 @@ async fn get_script_by_path_w_draft(
let mut tx = user_db.begin(&authed).await?;
let script_o = sqlx::query_as::<_, ScriptWDraft<ScriptRunnableSettingsHandle>>(
"SELECT hash, script.path, summary, description, content, language, kind, tag, schema, draft_only, envs, runnable_settings_handle, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, ws_error_handler_muted, draft.value as draft, dedicated_worker, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, has_preprocessor, on_behalf_of_email, assets, modules, debounce_key, debounce_delay_s, labels FROM script LEFT JOIN draft ON
"SELECT hash, script.path, summary, description, content, language, kind, tag, schema, draft_only, envs, runnable_settings_handle, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, ws_error_handler_muted, draft.value as draft, dedicated_worker, priority, restart_unless_cancelled, delete_after_use, timeout, concurrency_key, visible_to_runner_only, auto_kind, has_preprocessor, on_behalf_of_email, assets, modules, debounce_key, debounce_delay_s, labels FROM script LEFT JOIN draft ON
script.path = draft.path AND script.workspace_id = draft.workspace_id AND draft.typ = 'script'
WHERE script.path = $1 AND script.workspace_id = $2
ORDER BY script.created_at DESC LIMIT 1",

View File

@@ -1,157 +0,0 @@
//! Persistent, leased state for long-running background tasks coordinated
//! across multiple server replicas.
//!
//! Prior to this module, task state like "is log cleanup running?" lived in a
//! per-process `lazy_static` RwLock, which silently broke in multi-server
//! deployments: the server that claimed the lock was rarely the one the
//! frontend's GET hit next. This module stores state in the dedicated
//! `background_task_state` table and exposes a lease-based API:
//!
//! * [`try_claim`] atomically claims the named task, failing if another owner
//! currently holds it with a fresh heartbeat.
//! * [`update_state`] writes progress and refreshes the heartbeat (the caller
//! must be the owner; stale owners are ignored).
//! * [`release`] marks the task finished.
//! * [`get`] returns the current state. If the owner's heartbeat is stale,
//! the read returns `running = false` so other servers can reclaim.
//!
//! The stale-heartbeat threshold is [`STALE_HEARTBEAT_SECS`]. Tasks should
//! call [`update_state`] at least once every threshold/2 seconds.
use chrono::{DateTime, Utc};
use serde::Serialize;
use sqlx::{Pool, Postgres};
use windmill_common::error::{self, Result};
/// After this many seconds without a heartbeat, a "running" lease is treated
/// as stale and can be reclaimed by another server.
pub const STALE_HEARTBEAT_SECS: i64 = 120;
#[derive(Debug, Clone, Serialize)]
pub struct BackgroundTaskRow {
pub name: String,
pub value: serde_json::Value,
pub running: bool,
pub owner: Option<String>,
pub started_at: Option<DateTime<Utc>>,
pub finished_at: Option<DateTime<Utc>>,
pub updated_at: DateTime<Utc>,
}
/// Try to atomically claim the named task for this server.
///
/// Succeeds if:
/// * no row exists yet, or
/// * the existing row is not running, or
/// * the existing row's heartbeat (`updated_at`) is older than
/// [`STALE_HEARTBEAT_SECS`].
///
/// Returns `Ok(true)` if the caller now holds the lease, `Ok(false)` if
/// someone else is actively running it.
pub async fn try_claim<T: Serialize>(
db: &Pool<Postgres>,
name: &str,
owner: &str,
initial_value: &T,
) -> Result<bool> {
let value = serde_json::to_value(initial_value).map_err(|e| {
error::Error::internal_err(format!("serialize background_task initial value: {e}"))
})?;
let stale_cutoff = format!("{STALE_HEARTBEAT_SECS} seconds");
let claimed = sqlx::query_scalar!(
"INSERT INTO background_task_state
(name, value, running, owner, started_at, finished_at, updated_at)
VALUES ($1, $2, true, $3, NOW(), NULL, NOW())
ON CONFLICT (name) DO UPDATE SET
value = EXCLUDED.value,
running = true,
owner = EXCLUDED.owner,
started_at = NOW(),
finished_at = NULL,
updated_at = NOW()
WHERE background_task_state.running = false
OR background_task_state.updated_at < NOW() - ($4::text)::interval
RETURNING 1",
name,
value,
owner,
stale_cutoff,
)
.fetch_optional(db)
.await?;
Ok(claimed.is_some())
}
/// Update the task's stored state. Also refreshes the heartbeat.
///
/// No-ops if another owner has since claimed the lease (e.g. because our
/// heartbeat went stale and another server took over); callers should not
/// treat this as an error.
pub async fn update_state<T: Serialize>(
db: &Pool<Postgres>,
name: &str,
owner: &str,
value: &T,
) -> Result<()> {
let value = serde_json::to_value(value)
.map_err(|e| error::Error::internal_err(format!("serialize background_task value: {e}")))?;
sqlx::query!(
"UPDATE background_task_state
SET value = $1, updated_at = NOW()
WHERE name = $2 AND owner = $3 AND running = true",
value,
name,
owner
)
.execute(db)
.await?;
Ok(())
}
/// Mark the task as finished. Only the current owner can release.
pub async fn release<T: Serialize>(
db: &Pool<Postgres>,
name: &str,
owner: &str,
final_value: &T,
) -> Result<()> {
let value = serde_json::to_value(final_value).map_err(|e| {
error::Error::internal_err(format!("serialize background_task final value: {e}"))
})?;
sqlx::query!(
"UPDATE background_task_state
SET value = $1, running = false, finished_at = NOW(), updated_at = NOW()
WHERE name = $2 AND owner = $3",
value,
name,
owner
)
.execute(db)
.await?;
Ok(())
}
/// Read the current state of the named task.
///
/// If the row claims to be running but its heartbeat is stale (exceeds
/// [`STALE_HEARTBEAT_SECS`]), `running` is reported as `false` so the caller
/// / UI does not believe a dead server is still working.
pub async fn get(db: &Pool<Postgres>, name: &str) -> Result<Option<BackgroundTaskRow>> {
let row = sqlx::query_as!(
BackgroundTaskRow,
r#"SELECT name, value as "value!: serde_json::Value", running as "running!",
owner, started_at, finished_at, updated_at as "updated_at!"
FROM background_task_state WHERE name = $1"#,
name
)
.fetch_optional(db)
.await?;
Ok(row.map(|mut r| {
if r.running && r.updated_at < Utc::now() - chrono::Duration::seconds(STALE_HEARTBEAT_SECS)
{
r.running = false;
}
r
}))
}

View File

@@ -8,15 +8,9 @@
use std::{collections::HashMap, time::Duration};
#[cfg(feature = "parquet")]
mod background_task;
#[cfg(feature = "private")]
mod ee;
pub mod ee_oss;
#[cfg(feature = "parquet")]
mod log_cleanup;
#[cfg(feature = "parquet")]
mod storage_usage;
use windmill_api_auth::{require_devops_role, require_super_admin, ApiAuthed};
use windmill_common::utils::HTTP_CLIENT_PERMISSIVE as HTTP_CLIENT;
@@ -40,9 +34,7 @@ use serde::{Deserialize, Serialize};
#[cfg(feature = "enterprise")]
use windmill_common::ee_oss::{send_critical_alert, CriticalAlertKind, CriticalErrorChannel};
#[cfg(all(feature = "private", feature = "enterprise"))]
use windmill_common::secret_backend::{
AwsSecretsManagerSettings, AzureKeyVaultSettings, SecretMigrationReport, VaultSettings,
};
use windmill_common::secret_backend::{SecretMigrationReport, VaultSettings};
use windmill_common::{
ai_cache::bump_instance_ai_config_revision,
email_oss::send_email_plain_text,
@@ -101,10 +93,6 @@ pub fn global_service() -> Router {
"/setup_custom_instance_pg_database/{name}",
post(setup_custom_instance_pg_database),
)
.route(
"/drop_custom_instance_pg_database/{name}",
post(drop_custom_instance_pg_database),
)
.route(
"/critical_alerts/acknowledge_all",
post(acknowledge_all_critical_alerts),
@@ -118,7 +106,7 @@ pub fn global_service() -> Router {
post(restart_worker_group),
);
// Vault/Azure KV integration routes (EE only - requires both private and enterprise features)
// Vault integration routes (EE only - requires both private and enterprise features)
#[cfg(all(feature = "private", feature = "enterprise"))]
let r = r
.route("/test_secret_backend", post(test_secret_backend))
@@ -126,36 +114,11 @@ pub fn global_service() -> Router {
.route(
"/migrate_secrets_to_database",
post(migrate_secrets_to_database),
)
.route("/test_azure_kv_backend", post(test_azure_kv_backend))
.route(
"/migrate_secrets_to_azure_kv",
post(migrate_secrets_to_azure_kv),
)
.route(
"/migrate_secrets_from_azure_kv",
post(migrate_secrets_from_azure_kv),
)
.route("/test_aws_sm_backend", post(test_aws_sm_backend))
.route(
"/migrate_secrets_to_aws_sm",
post(migrate_secrets_to_aws_sm),
)
.route(
"/migrate_secrets_from_aws_sm",
post(migrate_secrets_from_aws_sm),
);
#[cfg(feature = "parquet")]
{
return r
.route("/test_object_storage_config", post(test_s3_bucket))
.route(
"/object_storage_usage",
get(get_object_storage_usage).post(compute_object_storage_usage),
)
.route("/run_log_cleanup", post(run_log_cleanup))
.route("/log_cleanup_status", get(log_cleanup_status));
return r.route("/test_object_storage_config", post(test_s3_bucket));
}
#[cfg(not(feature = "parquet"))]
@@ -253,46 +216,6 @@ pub async fn test_s3_bucket(
Ok("Tested blob storage successfully".to_string())
}
#[cfg(feature = "parquet")]
async fn get_object_storage_usage(
Extension(db): Extension<DB>,
authed: ApiAuthed,
) -> error::JsonResult<Option<storage_usage::StorageUsageProgress>> {
require_super_admin(&db, &authed.email).await?;
Ok(Json(storage_usage::get_status(&db).await?))
}
#[cfg(feature = "parquet")]
async fn compute_object_storage_usage(
Extension(db): Extension<DB>,
authed: ApiAuthed,
) -> error::Result<axum::http::StatusCode> {
require_super_admin(&db, &authed.email).await?;
storage_usage::try_start(&db).await?;
storage_usage::spawn_compute(db.clone());
Ok(axum::http::StatusCode::ACCEPTED)
}
#[cfg(feature = "parquet")]
async fn run_log_cleanup(
Extension(db): Extension<DB>,
authed: ApiAuthed,
) -> error::Result<axum::http::StatusCode> {
require_super_admin(&db, &authed.email).await?;
log_cleanup::try_start(&db).await?;
log_cleanup::spawn_cleanup(db.clone());
Ok(axum::http::StatusCode::ACCEPTED)
}
#[cfg(feature = "parquet")]
async fn log_cleanup_status(
Extension(db): Extension<DB>,
authed: ApiAuthed,
) -> error::JsonResult<Option<log_cleanup::LogCleanupProgress>> {
require_super_admin(&db, &authed.email).await?;
Ok(Json(log_cleanup::get_status(&db).await?))
}
#[derive(Deserialize)]
pub struct TestKey {
pub license_key: String,
@@ -1183,18 +1106,6 @@ async fn setup_custom_instance_pg_database_inner(
Ok(())
}
async fn drop_custom_instance_pg_database(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(dbname): Path<String>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
windmill_common::drop_custom_instance_database(&db, &dbname).await?;
Ok(format!("Database '{}' dropped successfully", dbname))
}
// ============================================================================
// Secret Backend Settings (HashiCorp Vault Integration) - Enterprise Edition
// ============================================================================
@@ -1259,93 +1170,6 @@ pub async fn migrate_secrets_to_database(
Ok(Json(report))
}
/// Test connection to Azure Key Vault
///
/// This is an Enterprise Edition feature.
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn test_azure_kv_backend(
Extension(db): Extension<DB>,
authed: ApiAuthed,
Json(settings): Json<AzureKeyVaultSettings>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
windmill_common::secret_backend::test_azure_kv_connection(&settings).await?;
Ok("Successfully connected to Azure Key Vault".to_string())
}
/// Migrate existing secrets from database to Azure Key Vault
///
/// This is an Enterprise Edition feature.
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn migrate_secrets_to_azure_kv(
Extension(db): Extension<DB>,
authed: ApiAuthed,
Json(settings): Json<AzureKeyVaultSettings>,
) -> JsonResult<SecretMigrationReport> {
require_super_admin(&db, &authed.email).await?;
let report =
windmill_common::secret_backend::migrate_secrets_to_azure_kv(&db, &settings).await?;
Ok(Json(report))
}
/// Migrate secrets from Azure Key Vault back to database
///
/// This is an Enterprise Edition feature.
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn migrate_secrets_from_azure_kv(
Extension(db): Extension<DB>,
authed: ApiAuthed,
Json(settings): Json<AzureKeyVaultSettings>,
) -> JsonResult<SecretMigrationReport> {
require_super_admin(&db, &authed.email).await?;
let report =
windmill_common::secret_backend::migrate_secrets_from_azure_kv(&db, &settings).await?;
Ok(Json(report))
}
/// Test connection to AWS Secrets Manager
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn test_aws_sm_backend(
Extension(db): Extension<DB>,
authed: ApiAuthed,
Json(settings): Json<AwsSecretsManagerSettings>,
) -> Result<String> {
require_super_admin(&db, &authed.email).await?;
windmill_common::secret_backend::test_aws_sm_connection(&settings).await?;
Ok("Successfully connected to AWS Secrets Manager".to_string())
}
/// Migrate existing secrets from database to AWS Secrets Manager
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn migrate_secrets_to_aws_sm(
Extension(db): Extension<DB>,
authed: ApiAuthed,
Json(settings): Json<AwsSecretsManagerSettings>,
) -> JsonResult<SecretMigrationReport> {
require_super_admin(&db, &authed.email).await?;
let report = windmill_common::secret_backend::migrate_secrets_to_aws_sm(&db, &settings).await?;
Ok(Json(report))
}
/// Migrate secrets from AWS Secrets Manager back to database
#[cfg(all(feature = "private", feature = "enterprise"))]
pub async fn migrate_secrets_from_aws_sm(
Extension(db): Extension<DB>,
authed: ApiAuthed,
Json(settings): Json<AwsSecretsManagerSettings>,
) -> JsonResult<SecretMigrationReport> {
require_super_admin(&db, &authed.email).await?;
let report =
windmill_common::secret_backend::migrate_secrets_from_aws_sm(&db, &settings).await?;
Ok(Json(report))
}
// ============================================================================
// JWKS Endpoint for Vault JWT Authentication
// ============================================================================

View File

@@ -1,683 +0,0 @@
#![cfg(feature = "parquet")]
/*
* Manual trigger for cleaning up expired log files from object storage.
*
* Mirrors the periodic cleanup done in backend/src/monitor.rs::delete_expired_items,
* but runs on demand from the UI with progress reporting and uses
* ObjectStore::delete_stream for batched S3 deletes (up to 1000 per request).
*
* Note: unlike the periodic cleanup (which only hits S3 when MONITOR_LOGS_ON_OBJECT_STORE
* is enabled), this manual path ALWAYS issues S3 deletes. That is intentional: operators
* who previously ran with the setting OFF may have orphan log files in their bucket and
* need a way to reclaim that space. Do not add a MONITOR_LOGS_ON_OBJECT_STORE guard here
* without first considering that use case.
*
* Progress state is persisted in the `background_task_state` table (see
* windmill-common/src/background_task.rs) so that any API server replica can serve
* the status endpoint — not just the one that happened to receive the POST.
*/
use std::sync::Arc;
use chrono::{DateTime, Utc};
use futures::stream::StreamExt;
use serde::{Deserialize, Serialize};
use tokio::sync::RwLock;
use uuid::Uuid;
use crate::background_task;
use windmill_common::error::{self};
use windmill_common::tracing_init::{LOGS_SERVICE, TMP_WINDMILL_LOGS_SERVICE};
use windmill_common::worker::WINDMILL_DIR;
use windmill_common::{DB, INSTANCE_NAME, JOB_RETENTION_SECS, SERVICE_LOG_RETENTION_SECS};
use windmill_object_store::object_store_reexports::{ObjectStore, Path as ObjectPath};
pub const TASK_NAME: &str = "log_cleanup";
const SERVICE_LOG_BATCH: i64 = 2_000;
const JOB_BATCH: i64 = 1_000;
/// Number of S3 paths to accumulate before issuing a batched DeleteObjects /
/// v2_job membership check during the orphan scan.
const ORPHAN_BATCH: usize = 1_000;
/// Flush orphan_scanned to the DB every N inspected objects. Per-object
/// writes would turn a TB-bucket scan into millions of DB round-trips.
const ORPHAN_SCAN_FLUSH_TICK: u64 = 1_000;
/// Maximum time between heartbeats during the orphan scan. Must stay below
/// `STALE_HEARTBEAT_SECS / 2` so that a slow S3 LIST (rate-limited providers
/// can take >2 minutes per 1000-object page) doesn't let another replica
/// mistakenly reclaim the lease and run a concurrent scan.
const ORPHAN_HEARTBEAT_SECS: u64 = 30;
#[derive(Clone, Serialize, Deserialize)]
pub struct LogCleanupProgress {
pub running: bool,
pub started_at: DateTime<Utc>,
pub finished_at: Option<DateTime<Utc>>,
/// Human-readable description of the current phase.
pub phase: String,
pub total_service: u64,
pub processed_service: u64,
pub total_jobs: u64,
pub processed_jobs: u64,
pub s3_deleted: u64,
/// Number of S3 objects inspected during the orphan scan phase.
pub orphans_scanned: u64,
/// Number of orphan S3 objects deleted (no corresponding DB row).
pub orphans_deleted: u64,
pub errors: u64,
pub last_error: Option<String>,
}
impl LogCleanupProgress {
fn new_running() -> Self {
Self {
running: true,
started_at: Utc::now(),
finished_at: None,
phase: "starting".to_string(),
total_service: 0,
processed_service: 0,
total_jobs: 0,
processed_jobs: 0,
s3_deleted: 0,
orphans_scanned: 0,
orphans_deleted: 0,
errors: 0,
last_error: None,
}
}
}
/// Per-task mutable state shared across the async helpers. Holds the DB handle
/// and owner string so every mutation can be persisted atomically.
struct Session {
db: DB,
owner: String,
progress: RwLock<LogCleanupProgress>,
}
impl Session {
async fn update<F: FnOnce(&mut LogCleanupProgress)>(&self, f: F) {
let snapshot = {
let mut p = self.progress.write().await;
f(&mut p);
p.clone()
};
if let Err(e) =
background_task::update_state(&self.db, TASK_NAME, &self.owner, &snapshot).await
{
tracing::warn!("log cleanup: failed to persist progress: {e:#}");
}
}
async fn set_phase(&self, phase: &str) {
self.update(|p| p.phase = phase.to_string()).await;
}
async fn record_error(&self, msg: String) {
tracing::error!("log cleanup: {msg}");
self.update(|p| {
p.errors = p.errors.saturating_add(1);
p.last_error = Some(msg);
})
.await;
}
async fn release(&self) {
let snapshot = {
let mut p = self.progress.write().await;
p.running = false;
p.finished_at = Some(Utc::now());
p.phase = "done".to_string();
p.clone()
};
if let Err(e) = background_task::release(&self.db, TASK_NAME, &self.owner, &snapshot).await
{
tracing::warn!("log cleanup: failed to release lease: {e:#}");
}
}
}
/// Try to atomically claim the cleanup lease. Returns Ok on success, or
/// Err if another server/process already holds a fresh lease.
pub async fn try_start(db: &DB) -> error::Result<()> {
let claimed = background_task::try_claim(
db,
TASK_NAME,
&*INSTANCE_NAME,
&LogCleanupProgress::new_running(),
)
.await?;
if !claimed {
return Err(error::Error::BadRequest(
"Log cleanup is already running".to_string(),
));
}
Ok(())
}
/// Fetch the current status from the DB. Any API server can call this.
pub async fn get_status(db: &DB) -> error::Result<Option<LogCleanupProgress>> {
let row = background_task::get(db, TASK_NAME).await?;
let Some(r) = row else { return Ok(None) };
match serde_json::from_value::<LogCleanupProgress>(r.value) {
Ok(mut p) => {
// background_task::get collapses `running` to false when the
// heartbeat is stale — mirror that into the returned struct.
p.running = r.running;
Ok(Some(p))
}
Err(e) => Err(error::Error::internal_err(format!(
"deserialize log cleanup progress: {e:#}"
))),
}
}
/// Delete the given object paths from S3 in batches (uses ObjectStore::delete_stream
/// which on S3 issues a single DeleteObjects request per 1000 paths).
async fn s3_bulk_delete(
store: &Arc<dyn ObjectStore>,
paths: Vec<ObjectPath>,
) -> (u64 /* deleted */, u64 /* errors */) {
let stream = futures::stream::iter(paths.into_iter().map(Ok)).boxed();
let mut deleted = 0u64;
let mut errors = 0u64;
let mut res = store.delete_stream(stream);
while let Some(r) = res.next().await {
match r {
Ok(_) => deleted += 1,
Err(e) => {
errors += 1;
tracing::warn!("log cleanup: failed to delete object: {e:#}");
}
}
}
(deleted, errors)
}
/// Delete the given relative paths from the local filesystem under `base_dir`.
async fn disk_bulk_delete(base_dir: &str, rel_paths: &[String]) {
let futs = rel_paths.iter().map(|p| async move {
let full = std::path::Path::new(base_dir).join(p);
if tokio::fs::metadata(&full).await.is_ok() {
if let Err(e) = tokio::fs::remove_file(&full).await {
tracing::warn!(
"log cleanup: failed to delete {}: {e}",
full.to_string_lossy()
);
}
}
});
futures::future::join_all(futs).await;
}
async fn cleanup_service_logs(
session: &Session,
db: &DB,
store: &Arc<dyn ObjectStore>,
) -> error::Result<()> {
// Count candidates upfront for progress reporting.
let total: i64 = sqlx::query_scalar!(
"SELECT COUNT(*) FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval",
SERVICE_LOG_RETENTION_SECS,
)
.fetch_one(db)
.await?
.unwrap_or(0);
session.update(|p| p.total_service = total as u64).await;
if total <= 0 {
return Ok(());
}
struct LogFileRow {
file_path: String,
hostname: String,
}
loop {
let rows = sqlx::query_as!(
LogFileRow,
"DELETE FROM log_file WHERE (file_path, hostname) IN (
SELECT file_path, hostname FROM log_file
WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval
LIMIT $2
) RETURNING file_path, hostname",
SERVICE_LOG_RETENTION_SECS,
SERVICE_LOG_BATCH,
)
.fetch_all(db)
.await?;
if rows.is_empty() {
break;
}
let rel_paths: Vec<String> = rows
.iter()
.map(|r| format!("{}/{}", r.hostname, r.file_path))
.collect();
let batch_len = rel_paths.len() as u64;
let s3_paths: Vec<ObjectPath> = rel_paths
.iter()
.map(|p| ObjectPath::from(format!("{}{}", LOGS_SERVICE, p)))
.collect();
let (deleted, errors) = s3_bulk_delete(store, s3_paths).await;
disk_bulk_delete(&*TMP_WINDMILL_LOGS_SERVICE, &rel_paths).await;
session
.update(|p| {
p.processed_service = p.processed_service.saturating_add(batch_len);
if p.processed_service > p.total_service {
p.total_service = p.processed_service;
}
p.s3_deleted = p.s3_deleted.saturating_add(deleted);
p.errors = p.errors.saturating_add(errors);
})
.await;
}
// Collapse the total to what we actually processed.
session
.update(|p| p.total_service = p.processed_service)
.await;
Ok(())
}
async fn cleanup_job_logs(
session: &Session,
db: &DB,
store: &Arc<dyn ObjectStore>,
) -> error::Result<()> {
let retention_secs = *JOB_RETENTION_SECS.read().await;
if retention_secs <= 0 {
return Ok(());
}
let total: i64 = sqlx::query_scalar!(
"SELECT COUNT(*) FROM v2_job_completed
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval",
retention_secs,
)
.fetch_one(db)
.await?
.unwrap_or(0);
session.update(|p| p.total_jobs = total as u64).await;
if total <= 0 {
return Ok(());
}
loop {
let (deleted_count, rel_paths) =
delete_expired_jobs_batch(db, retention_secs, JOB_BATCH).await?;
if deleted_count == 0 {
break;
}
let s3_paths: Vec<ObjectPath> = rel_paths
.iter()
.map(|p| ObjectPath::from(p.clone()))
.collect();
let (deleted, errors) = s3_bulk_delete(store, s3_paths).await;
disk_bulk_delete(&*WINDMILL_DIR, &rel_paths).await;
session
.update(|p| {
p.processed_jobs = p.processed_jobs.saturating_add(deleted_count as u64);
if p.processed_jobs > p.total_jobs {
p.total_jobs = p.processed_jobs;
}
p.s3_deleted = p.s3_deleted.saturating_add(deleted);
p.errors = p.errors.saturating_add(errors);
})
.await;
}
// Collapse the total to what we actually processed — the upfront count
// includes jobs whose root is still active (protected from deletion), so
// without this the progress bar would get stuck at e.g. 3/44.
session.update(|p| p.total_jobs = p.processed_jobs).await;
Ok(())
}
/// Mirrors backend/src/monitor.rs::delete_expired_jobs_batch but returns the
/// log paths instead of deleting them from storage itself, so the caller can
/// issue a single batched S3 delete across many batches via delete_stream.
async fn delete_expired_jobs_batch(
db: &DB,
job_retention_secs: i64,
batch_size: i64,
) -> error::Result<(usize, Vec<String>)> {
let mut tx = db.begin().await?;
let active_root_job_ids: Vec<Uuid> = sqlx::query_scalar!(
"SELECT q.id FROM v2_job_queue q
JOIN v2_job j ON j.id = q.id
WHERE j.parent_job IS NULL
AND j.created_at <= now() - ($1::bigint::text || ' s')::interval",
job_retention_secs
)
.fetch_all(&mut *tx)
.await?;
let deleted_jobs: Vec<Uuid> = sqlx::query_scalar!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT jc.id FROM v2_job_completed jc
LEFT JOIN v2_job j ON j.id = jc.id
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)
ORDER BY jc.completed_at ASC
LIMIT $2
FOR UPDATE OF jc SKIP LOCKED
)
RETURNING id",
job_retention_secs,
batch_size,
&active_root_job_ids
)
.fetch_all(&mut *tx)
.await?;
let deleted_count = deleted_jobs.len();
if deleted_count == 0 {
tx.commit().await?;
return Ok((0, Vec::new()));
}
if let Err(e) = sqlx::query!(
"DELETE FROM job_stats WHERE job_id = ANY($1)",
&deleted_jobs
)
.execute(&mut *tx)
.await
{
tracing::error!("log cleanup: error deleting job stats: {e:?}");
}
let log_paths: Vec<String> = match sqlx::query_scalar!(
"DELETE FROM job_logs WHERE job_id = ANY($1) RETURNING log_file_index",
&deleted_jobs
)
.fetch_all(&mut *tx)
.await
{
Ok(log_file_index) => log_file_index
.into_iter()
.filter_map(|opt| opt)
.flat_map(|inner_vec| inner_vec.into_iter())
.collect(),
Err(e) => {
tracing::error!("log cleanup: error deleting job logs: {e:?}");
Vec::new()
}
};
if let Err(e) = sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", &deleted_jobs)
.execute(&mut *tx)
.await
{
tracing::error!("log cleanup: error deleting job: {e:?}");
}
if let Err(e) = sqlx::query!(
"DELETE FROM job_result_stream_v2 WHERE job_id = ANY($1)",
&deleted_jobs
)
.execute(&mut *tx)
.await
{
tracing::error!("log cleanup: error deleting job result stream: {e:?}");
}
tx.commit().await?;
Ok((deleted_count, log_paths))
}
/// Scan S3 under the `logs/` prefix for orphan log files and delete them.
///
/// An orphan is an S3 object that is older than retention and has no corresponding
/// live job / service-log row in the DB. This is how we reclaim space from:
/// - operators who previously ran with MONITOR_LOGS_ON_OBJECT_STORE off (so DB
/// cleanup skipped S3 and left files behind when jobs were deleted)
/// - previous failed batched S3 deletes that left stragglers
///
/// Job log paths look like `logs/<uuid>/<ts>_<size>.txt` — we parse the uuid and
/// protect any object whose job_id still exists in `v2_job` (both queued and
/// completed jobs, so active flows keep their logs until their root completes
/// and regular DB-driven cleanup catches them).
///
/// Service log paths look like `logs/services/<hostname>/<file>` — we cannot
/// reliably map them to `log_file` rows (file_path format is rotation-dependent)
/// so we rely on `last_modified` alone, which after the DB phase completed
/// already reflects true orphans.
async fn cleanup_s3_orphans(
session: &Session,
db: &DB,
store: &Arc<dyn ObjectStore>,
) -> error::Result<()> {
let job_retention_secs = *JOB_RETENTION_SECS.read().await;
let now = Utc::now();
// Service logs always have a retention (hardcoded SERVICE_LOG_RETENTION_SECS),
// so we scan for service-log orphans regardless of JOB_RETENTION_SECS. Job-log
// orphans, by contrast, can only be considered expired relative to
// JOB_RETENTION_SECS; when that is disabled we skip the job branch entirely.
let service_cutoff = now - chrono::Duration::seconds(SERVICE_LOG_RETENTION_SECS);
let job_cutoff = if job_retention_secs > 0 {
Some(now - chrono::Duration::seconds(job_retention_secs))
} else {
None
};
let logs_prefix = ObjectPath::from("logs/");
let mut stream = store.list(Some(&logs_prefix));
let mut service_batch: Vec<ObjectPath> = Vec::with_capacity(ORPHAN_BATCH);
let mut job_batch: Vec<(ObjectPath, Uuid)> = Vec::with_capacity(ORPHAN_BATCH);
let mut scanned_since_flush: u64 = 0;
let mut last_heartbeat = std::time::Instant::now();
while let Some(item) = stream.next().await {
let meta = match item {
Ok(m) => m,
Err(e) => {
session.record_error(format!("list objects: {e:#}")).await;
// Listing error aborts (continuation token is gone); the next
// manual run will pick up where we left off.
break;
}
};
scanned_since_flush += 1;
if scanned_since_flush >= ORPHAN_SCAN_FLUSH_TICK
|| last_heartbeat.elapsed() >= std::time::Duration::from_secs(ORPHAN_HEARTBEAT_SECS)
{
let delta = scanned_since_flush;
scanned_since_flush = 0;
last_heartbeat = std::time::Instant::now();
session
.update(|p| p.orphans_scanned = p.orphans_scanned.saturating_add(delta))
.await;
}
let path_str = meta.location.as_ref();
let rest = match path_str.strip_prefix("logs/") {
Some(r) => r,
None => continue,
};
if let Some(_after_services) = rest.strip_prefix("services/") {
if meta.last_modified < service_cutoff {
service_batch.push(meta.location);
if service_batch.len() >= ORPHAN_BATCH {
flush_service_orphans(session, store, &mut service_batch).await;
}
}
} else if let Some(job_cutoff) = job_cutoff {
// logs/<uuid>/... — parse uuid segment.
let first_seg = rest.split('/').next().unwrap_or("");
let job_id = match Uuid::parse_str(first_seg) {
Ok(u) => u,
Err(_) => continue,
};
if meta.last_modified < job_cutoff {
job_batch.push((meta.location, job_id));
if job_batch.len() >= ORPHAN_BATCH {
flush_job_orphans(session, db, store, &mut job_batch).await;
}
}
}
}
// Flush remaining progress counter and any residual delete batches.
if scanned_since_flush > 0 {
let delta = scanned_since_flush;
session
.update(|p| p.orphans_scanned = p.orphans_scanned.saturating_add(delta))
.await;
}
if !service_batch.is_empty() {
flush_service_orphans(session, store, &mut service_batch).await;
}
if !job_batch.is_empty() {
flush_job_orphans(session, db, store, &mut job_batch).await;
}
Ok(())
}
async fn flush_service_orphans(
session: &Session,
store: &Arc<dyn ObjectStore>,
batch: &mut Vec<ObjectPath>,
) {
let paths = std::mem::take(batch);
let (deleted, errors) = s3_bulk_delete(store, paths).await;
session
.update(|p| {
p.orphans_deleted = p.orphans_deleted.saturating_add(deleted);
p.s3_deleted = p.s3_deleted.saturating_add(deleted);
p.errors = p.errors.saturating_add(errors);
})
.await;
}
async fn flush_job_orphans(
session: &Session,
db: &DB,
store: &Arc<dyn ObjectStore>,
batch: &mut Vec<(ObjectPath, Uuid)>,
) {
let taken = std::mem::take(batch);
let ids: Vec<Uuid> = {
let mut seen = std::collections::HashSet::with_capacity(taken.len());
taken
.iter()
.filter_map(|(_, id)| if seen.insert(*id) { Some(*id) } else { None })
.collect()
};
let protected: std::collections::HashSet<Uuid> =
match sqlx::query_scalar!("SELECT id FROM v2_job WHERE id = ANY($1)", &ids)
.fetch_all(db)
.await
{
Ok(rows) => rows.into_iter().collect(),
Err(e) => {
session
.record_error(format!("orphan membership check failed: {e:#}"))
.await;
return;
}
};
let to_delete: Vec<ObjectPath> = taken
.into_iter()
.filter_map(|(path, id)| {
if protected.contains(&id) {
None
} else {
Some(path)
}
})
.collect();
if to_delete.is_empty() {
return;
}
let (deleted, errors) = s3_bulk_delete(store, to_delete).await;
session
.update(|p| {
p.orphans_deleted = p.orphans_deleted.saturating_add(deleted);
p.s3_deleted = p.s3_deleted.saturating_add(deleted);
p.errors = p.errors.saturating_add(errors);
})
.await;
}
/// Spawn the cleanup task. Caller is responsible for ensuring only one runs at a time
/// (use `try_start` first).
pub fn spawn_cleanup(db: DB) {
use futures::FutureExt;
use std::panic::AssertUnwindSafe;
tokio::spawn(async move {
let session = Arc::new(Session {
db: db.clone(),
owner: INSTANCE_NAME.clone(),
progress: RwLock::new(LogCleanupProgress::new_running()),
});
let s = session.clone();
let task = async move {
let store = match windmill_object_store::get_object_store().await {
Some(st) => st,
None => {
s.record_error("Object storage is not configured".to_string())
.await;
return;
}
};
s.set_phase("service logs (db)").await;
if let Err(e) = cleanup_service_logs(&s, &db, &store).await {
s.record_error(format!("service logs phase failed: {e:#}"))
.await;
}
s.set_phase("job logs (db)").await;
if let Err(e) = cleanup_job_logs(&s, &db, &store).await {
s.record_error(format!("job logs phase failed: {e:#}"))
.await;
}
s.set_phase("orphan S3 scan").await;
if let Err(e) = cleanup_s3_orphans(&s, &db, &store).await {
s.record_error(format!("orphan scan phase failed: {e:#}"))
.await;
}
};
// catch_unwind so a panic inside the cleanup can't leave the lease held forever.
if let Err(panic) = AssertUnwindSafe(task).catch_unwind().await {
let msg = panic
.downcast_ref::<&str>()
.map(|s| s.to_string())
.or_else(|| panic.downcast_ref::<String>().cloned())
.unwrap_or_else(|| "unknown panic".to_string());
session
.record_error(format!("cleanup task panicked: {msg}"))
.await;
}
session.release().await;
});
}

View File

@@ -1,256 +0,0 @@
#![cfg(feature = "parquet")]
/*
* Background computation of object-storage usage aggregated by top-level folder.
*
* On large buckets (millions of objects, TB of data) `list` is an O(N) operation:
* the object store returns 1000 objects per API call, so summing sizes under
* "logs/" can take several minutes. That is far longer than most reverse proxies
* allow in front of a single HTTP request (typically 30-60s), so we run the
* computation in a background tokio task and expose it via a status endpoint
* that the frontend polls.
*
* Progress is persisted in the `background_task_state` table so the status
* endpoint works across API-server replicas (the task runs on whichever
* server received the POST; any server can serve the GET).
*/
use std::sync::Arc;
use chrono::{DateTime, Utc};
use futures::stream::StreamExt;
use serde::{Deserialize, Serialize};
use tokio::sync::RwLock;
use crate::background_task;
use windmill_common::error;
use windmill_common::{DB, INSTANCE_NAME};
use windmill_object_store::object_store_reexports::ObjectStore;
pub const TASK_NAME: &str = "storage_usage";
#[derive(Clone, Serialize, Deserialize)]
pub struct FolderUsage {
pub prefix: String,
pub size: u64,
/// True if listing this prefix errored mid-stream; `size` is then a lower
/// bound, not the full total. UI surfaces this so operators don't
/// misinterpret the partial sum as the real folder size.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub partial: bool,
}
#[derive(Clone, Serialize, Deserialize)]
pub struct StorageUsageProgress {
pub running: bool,
pub started_at: DateTime<Utc>,
pub finished_at: Option<DateTime<Utc>>,
pub current_prefix: Option<String>,
pub scanned_objects: u64,
pub folders: Vec<FolderUsage>,
pub error: Option<String>,
}
impl StorageUsageProgress {
fn new_running() -> Self {
Self {
running: true,
started_at: Utc::now(),
finished_at: None,
current_prefix: None,
scanned_objects: 0,
folders: Vec::new(),
error: None,
}
}
}
struct Session {
db: DB,
owner: String,
progress: RwLock<StorageUsageProgress>,
}
impl Session {
async fn update<F: FnOnce(&mut StorageUsageProgress)>(&self, f: F) {
let snapshot = {
let mut p = self.progress.write().await;
f(&mut p);
p.clone()
};
if let Err(e) =
background_task::update_state(&self.db, TASK_NAME, &self.owner, &snapshot).await
{
tracing::warn!("storage usage: failed to persist progress: {e:#}");
}
}
async fn release(&self) {
let snapshot = {
let mut p = self.progress.write().await;
p.running = false;
p.finished_at = Some(Utc::now());
p.clone()
};
if let Err(e) = background_task::release(&self.db, TASK_NAME, &self.owner, &snapshot).await
{
tracing::warn!("storage usage: failed to release lease: {e:#}");
}
}
}
pub async fn try_start(db: &DB) -> error::Result<()> {
let claimed = background_task::try_claim(
db,
TASK_NAME,
&*INSTANCE_NAME,
&StorageUsageProgress::new_running(),
)
.await?;
if !claimed {
return Err(error::Error::BadRequest(
"Storage usage computation is already running".to_string(),
));
}
Ok(())
}
pub async fn get_status(db: &DB) -> error::Result<Option<StorageUsageProgress>> {
let row = background_task::get(db, TASK_NAME).await?;
let Some(r) = row else { return Ok(None) };
match serde_json::from_value::<StorageUsageProgress>(r.value) {
Ok(mut p) => {
p.running = r.running;
Ok(Some(p))
}
Err(e) => Err(error::Error::internal_err(format!(
"deserialize storage usage progress: {e:#}"
))),
}
}
async fn compute_inner(session: &Session, store: Arc<dyn ObjectStore>) -> error::Result<()> {
let list_result = store
.list_with_delimiter(None)
.await
.map_err(|e| error::Error::internal_err(format!("list_with_delimiter at root: {e:#}")))?;
let root_size: u64 = list_result.objects.iter().map(|o| o.size).sum();
if root_size > 0 {
let root_count = list_result.objects.len() as u64;
session
.update(|p| {
p.folders.push(FolderUsage {
prefix: "(root files)".to_string(),
size: root_size,
partial: false,
});
p.scanned_objects = p.scanned_objects.saturating_add(root_count);
})
.await;
}
/// Keep heartbeat well below STALE_HEARTBEAT_SECS/2 so a slow S3 LIST
/// (large bucket, rate-limited) can't let another replica reclaim.
const HEARTBEAT_SECS: u64 = 30;
const PROGRESS_TICK: u64 = 1_000;
for prefix in list_result.common_prefixes {
let prefix_str = prefix.to_string();
session
.update(|p| p.current_prefix = Some(prefix_str.clone()))
.await;
let mut total_size: u64 = 0;
let mut local_count: u64 = 0;
let mut partial = false;
let mut last_heartbeat = std::time::Instant::now();
let mut stream = store.list(Some(&prefix));
while let Some(item) = stream.next().await {
match item {
Ok(meta) => {
total_size += meta.size;
local_count += 1;
if local_count % PROGRESS_TICK == 0
|| last_heartbeat.elapsed()
>= std::time::Duration::from_secs(HEARTBEAT_SECS)
{
last_heartbeat = std::time::Instant::now();
session
.update(|p| {
p.scanned_objects = p.scanned_objects.saturating_add(PROGRESS_TICK)
})
.await;
}
}
Err(e) => {
tracing::warn!("storage usage: error listing {prefix_str}: {e:#}");
partial = true;
break;
}
}
}
let remainder = local_count % PROGRESS_TICK;
if remainder > 0 {
session
.update(|p| p.scanned_objects = p.scanned_objects.saturating_add(remainder))
.await;
}
session
.update(|p| {
p.folders
.push(FolderUsage { prefix: prefix_str, size: total_size, partial });
// Keep folders sorted by size so UI stays stable as we stream results.
p.folders.sort_by(|a, b| b.size.cmp(&a.size));
})
.await;
}
session.update(|p| p.current_prefix = None).await;
Ok(())
}
pub fn spawn_compute(db: DB) {
use futures::FutureExt;
use std::panic::AssertUnwindSafe;
tokio::spawn(async move {
let session = Arc::new(Session {
db: db.clone(),
owner: INSTANCE_NAME.clone(),
progress: RwLock::new(StorageUsageProgress::new_running()),
});
let s = session.clone();
let task = async move {
let store = match windmill_object_store::get_object_store().await {
Some(st) => st,
None => {
s.update(|p| p.error = Some("Object storage is not configured".to_string()))
.await;
return;
}
};
if let Err(e) = compute_inner(&s, store).await {
s.update(|p| p.error = Some(format!("{e:#}"))).await;
}
};
if let Err(panic) = AssertUnwindSafe(task).catch_unwind().await {
let msg = panic
.downcast_ref::<&str>()
.map(|s| s.to_string())
.or_else(|| panic.downcast_ref::<String>().cloned())
.unwrap_or_else(|| "unknown panic".to_string());
session
.update(|p| p.error = Some(format!("task panicked: {msg}")))
.await;
}
session.release().await;
});
}

View File

@@ -27,7 +27,6 @@ windmill-git-sync.workspace = true
windmill-queue.workspace = true
windmill-api-jobs.workspace = true
windmill-dep-map.workspace = true
windmill-store.workspace = true
axum.workspace = true
chrono.workspace = true

View File

@@ -40,8 +40,8 @@ use windmill_common::workspaces::GitRepositorySettings;
use windmill_common::workspaces::WorkspaceDeploymentUISettings;
use windmill_common::workspaces::{
check_user_against_rule, get_datatable_resource_from_db_unchecked, DataTable,
DataTableCatalogResourceType, DataTableForkBehavior, ProtectionRuleKind, ProtectionRules,
ProtectionRuleset, RuleCheckResult, WorkspaceGitSyncSettings,
DataTableCatalogResourceType, ProtectionRuleKind, ProtectionRules, ProtectionRuleset,
RuleCheckResult, WorkspaceGitSyncSettings,
};
use windmill_common::workspaces::{Ducklake, DucklakeCatalogResourceType};
use windmill_common::PgDatabase;
@@ -157,17 +157,6 @@ pub fn workspaced_service() -> Router {
post(reset_workspace_diffs),
)
.route("/compare/{target_workspace_id}", get(compare_workspaces))
.route("/create_pg_database", post(create_pg_database))
.route("/import_pg_database", post(import_pg_database))
.route("/export_pg_schema", post(export_pg_schema))
.route(
"/drop_forked_datatable_databases",
post(crate::workspaces_extra::drop_forked_datatable_databases),
)
.route(
"/get_datatable_full_schema",
post(get_datatable_full_schema),
)
.route("/protection_rules", get(list_protection_rules))
.route("/protection_rules", post(create_protection_rule))
.route(
@@ -392,16 +381,6 @@ struct CreateWorkspaceFork {
id: String,
name: String,
color: Option<String>,
/// Datatable names that were forked. For each, the backend will update the
/// forked workspace's datatable config to point to the new database.
#[serde(default)]
forked_datatables: Vec<ForkedDatatableInfo>,
}
#[derive(Deserialize)]
struct ForkedDatatableInfo {
name: String,
new_dbname: String,
}
#[derive(Deserialize)]
@@ -1294,40 +1273,26 @@ async fn list_ducklakes(
Ok(Json(ducklakes))
}
#[derive(Serialize)]
struct DataTableListItem {
name: String,
resource_type: String,
resource_path: String,
}
async fn list_datatables(
_authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<Vec<DataTableListItem>> {
let config = sqlx::query_scalar!(
"SELECT datatable->'datatables' FROM workspace_settings WHERE workspace_id = $1",
) -> JsonResult<Vec<String>> {
let datatables = sqlx::query_scalar!(
r#"
SELECT jsonb_object_keys(ws.datatable->'datatables') AS datatable_name
FROM workspace_settings ws
WHERE ws.workspace_id = $1
"#,
&w_id
)
.fetch_one(&db)
.await?;
.fetch_all(&db)
.await?
.into_iter()
.filter_map(|s| s)
.collect();
let items: Vec<DataTableListItem> = match config {
Some(val) => {
let map: HashMap<String, DataTable> = serde_json::from_value(val).unwrap_or_default();
map.into_iter()
.map(|(name, dt)| DataTableListItem {
name,
resource_type: dt.database.resource_type.as_ref().to_string(),
resource_path: dt.database.resource_path,
})
.collect()
}
None => vec![],
};
Ok(Json(items))
Ok(Json(datatables))
}
/// Compact column representation: "type" or "type?" for nullable, with "=default" suffix if has default
@@ -1490,397 +1455,6 @@ async fn get_datatable_schema(db: &DB, w_id: &str, datatable_name: &str) -> Resu
Ok(schema_map)
}
/// Resolve a source string to PgDatabase credentials with user-scoped permission checks.
/// For `datatable://name`: accessible to everyone (variables are resolved internally).
/// For `$res:path`: uses UserDB (row-level security) to verify the user can see the resource,
/// then interpolates `$var:` references in the resource value.
pub(crate) async fn resolve_pg_source_checked(
db: &DB,
user_db: &UserDB,
authed: &ApiAuthed,
w_id: &str,
source: &str,
) -> Result<PgDatabase> {
let db_resource = if let Some(name) = source.strip_prefix("datatable://") {
get_datatable_resource_from_db_unchecked(db, w_id, name).await?
} else if let Some(path) = source.strip_prefix("$res:") {
let db_with_authed = windmill_common::db::DbWithOptAuthed::from_authed(
authed,
db.clone(),
Some(user_db.clone()),
);
let value = windmill_store::resources::get_resource_value_interpolated_internal(
&db_with_authed,
w_id,
path,
None,
None,
false,
)
.await?;
match value {
Some(v) => v,
None => {
return Err(Error::NotAuthorized(format!(
"Resource '{}' not found or you do not have access to it",
path
)));
}
}
} else {
return Err(Error::BadRequest(format!(
"Invalid source format: '{}'. Expected 'datatable://name' or '$res:path'",
source
)));
};
serde_json::from_value(db_resource)
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))
}
/// A temporary file for pg_dump output that is automatically deleted when dropped.
struct DumpFile {
path: std::path::PathBuf,
}
impl DumpFile {
fn new() -> Result<Self> {
let dir = std::path::Path::new("/tmp/windmill");
std::fs::create_dir_all(dir)
.map_err(|e| Error::internal_err(format!("Failed to create /tmp/windmill: {}", e)))?;
// Set directory permissions to owner-only
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let _ = std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700));
}
let path = dir.join(format!("datatable_dump_{}", uuid::Uuid::new_v4()));
// Create the file with restrictive permissions before pg_dump writes to it
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
std::fs::OpenOptions::new()
.write(true)
.create(true)
.mode(0o600)
.open(&path)
.map_err(|e| Error::internal_err(format!("Failed to create dump file: {}", e)))?;
}
#[cfg(not(unix))]
{
std::fs::File::create(&path)
.map_err(|e| Error::internal_err(format!("Failed to create dump file: {}", e)))?;
}
Ok(Self { path })
}
}
impl Drop for DumpFile {
fn drop(&mut self) {
if self.path.exists() {
if let Err(e) = std::fs::remove_file(&self.path) {
tracing::warn!("Failed to remove dump file {:?}: {}", self.path, e);
}
}
}
}
/// Run pg_dump against a PgDatabase, writing output to a temp file on disk.
/// Returns a DumpFile handle; the file is deleted when the handle is dropped.
async fn pg_dump_database(pg_db: &PgDatabase, schema_only: bool) -> Result<DumpFile> {
let dump_file = DumpFile::new()?;
let host = &pg_db.host;
let port = pg_db.port.unwrap_or(5432).to_string();
let user = pg_db.user.as_deref().unwrap_or("postgres");
let dbname = &pg_db.dbname;
let mut cmd = tokio::process::Command::new("pg_dump");
cmd.arg("--format=plain").arg("--file").arg(&dump_file.path);
if schema_only {
cmd.arg("--schema-only");
}
cmd.arg("--host")
.arg(host)
.arg("--port")
.arg(&port)
.arg("--username")
.arg(user)
.arg(dbname);
if let Some(ref password) = pg_db.password {
cmd.env("PGPASSWORD", password);
}
if let Some(ref sslmode) = pg_db.sslmode {
cmd.env("PGSSLMODE", sslmode);
}
let output = cmd
.output()
.await
.map_err(|e| Error::internal_err(format!("Failed to execute pg_dump: {}", e)))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(Error::internal_err(format!("pg_dump failed: {}", stderr)));
}
Ok(dump_file)
}
/// Import a pg_dump file into a target database using psql.
async fn pg_import_dump(target_db: &PgDatabase, dump_file: &DumpFile) -> Result<()> {
let host = &target_db.host;
let port = target_db.port.unwrap_or(5432).to_string();
let user = target_db.user.as_deref().unwrap_or("postgres");
let dbname = &target_db.dbname;
let mut cmd = tokio::process::Command::new("psql");
cmd.arg("--host")
.arg(host)
.arg("--port")
.arg(&port)
.arg("--username")
.arg(user)
.arg("--dbname")
.arg(dbname)
.arg("--no-psqlrc")
.arg("--file")
.arg(&dump_file.path)
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped());
if let Some(ref password) = target_db.password {
cmd.env("PGPASSWORD", password);
}
if let Some(ref sslmode) = target_db.sslmode {
cmd.env("PGSSLMODE", sslmode);
}
let output = cmd
.output()
.await
.map_err(|e| Error::internal_err(format!("Failed to execute psql: {}", e)))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(Error::internal_err(format!(
"psql import failed: {}",
stderr
)));
}
Ok(())
}
#[derive(Deserialize)]
struct CreatePgDatabaseRequest {
/// The datatable source to determine connection info: 'datatable://name' or '$res:path'
source: String,
/// Name for the new database
target_dbname: String,
}
/// Create a new PostgreSQL database. For instance datatables, creates on the Windmill PG instance.
/// For resource datatables, creates on the same server as the source.
async fn create_pg_database(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(req): Json<CreatePgDatabaseRequest>,
) -> Result<String> {
windmill_common::validate_dbname(&req.target_dbname)?;
// Non-superadmin: restrict dbname to wm_fork_ prefix
if !windmill_common::auth::is_super_admin_email(&db, &authed.email).await? {
if !req.target_dbname.starts_with("wm_fork_") {
return Err(Error::BadRequest(
"Non-superadmin users can only create databases with names starting with 'wm_fork_'"
.to_string(),
));
}
}
// Determine if this is an instance or resource-backed datatable
let is_instance_datatable = if let Some(dt_name) = req.source.strip_prefix("datatable://") {
let config = sqlx::query_scalar!(
"SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
&w_id,
dt_name
)
.fetch_optional(&db)
.await?
.flatten();
config
.and_then(|v| {
v.get("database")
.and_then(|d| d.get("resource_type"))
.and_then(|r| r.as_str())
.map(|s| s == "instance")
})
.unwrap_or(false)
} else {
false
};
if is_instance_datatable {
windmill_common::create_custom_instance_database(&db, &req.target_dbname, "datatable")
.await?;
} else {
let source_pg =
resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let (client, connection) = source_pg.connect().await?;
let join_handle = tokio::spawn(async move { connection.await });
let row = client
.query_one(
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_database WHERE datname = $1)",
&[&req.target_dbname],
)
.await
.map_err(|e| {
Error::internal_err(format!("Failed to check database existence: {}", e))
})?;
let db_exists: bool = row.get(0);
if db_exists {
drop(client);
let _ = join_handle.await;
return Err(Error::BadRequest(format!(
"Database '{}' already exists on the resource server",
req.target_dbname
)));
}
client
.execute(&format!("CREATE DATABASE \"{}\"", &req.target_dbname), &[])
.await
.map_err(|e| {
Error::internal_err(format!(
"Failed to create database '{}': {}",
req.target_dbname, e
))
})?;
drop(client);
join_handle
.await
.map_err(|e| Error::internal_err(format!("join error: {}", e)))?
.map_err(|e| Error::internal_err(format!("tokio_postgres error: {}", e)))?;
}
Ok(format!("Created database '{}'", req.target_dbname))
}
#[derive(Deserialize)]
struct ImportPgDatabaseRequest {
source: String,
target: String,
#[serde(default)]
target_dbname_override: Option<String>,
fork_behavior: DataTableForkBehavior,
}
/// Import (pg_dump/pg_import) from source to target
async fn import_pg_database(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(req): Json<ImportPgDatabaseRequest>,
) -> Result<String> {
if req.fork_behavior == DataTableForkBehavior::KeepOriginal {
return Ok("No action needed for KeepOriginal behavior".to_string());
}
if req.fork_behavior == DataTableForkBehavior::SchemaAndData {
require_admin(authed.is_admin, &authed.username)?;
if *CLOUD_HOSTED {
return Err(Error::BadRequest(
"Importing schema and data is not available on cloud".to_string(),
));
}
}
let schema_only = req.fork_behavior == DataTableForkBehavior::SchemaOnly;
let source_pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let mut target_pg =
resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.target).await?;
if let Some(ref override_dbname) = req.target_dbname_override {
if !windmill_common::auth::is_super_admin_email(&db, &authed.email).await? {
if !override_dbname.starts_with("wm_fork_") {
return Err(Error::BadRequest(
"Non-superadmin users can only override target dbname with names starting with 'wm_fork_'"
.to_string(),
));
}
}
target_pg.dbname = override_dbname.clone();
}
windmill_common::validate_dbname(&target_pg.dbname)?;
let dump_file = pg_dump_database(&source_pg, schema_only).await?;
pg_import_dump(&target_pg, &dump_file).await?;
Ok(format!(
"Imported from '{}' into '{}'",
req.source, target_pg.dbname
))
}
#[derive(Deserialize)]
struct ExportPgSchemaRequest {
source: String,
}
async fn export_pg_schema(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(req): Json<ExportPgSchemaRequest>,
) -> Result<String> {
let pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let dump_file = pg_dump_database(&pg, true).await?;
tokio::fs::read_to_string(&dump_file.path)
.await
.map_err(|e| Error::internal_err(format!("Failed to read dump file: {}", e)))
}
#[derive(Deserialize)]
struct GetDatatableFullSchemaRequest {
source: String,
}
async fn get_datatable_full_schema(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(req): Json<GetDatatableFullSchemaRequest>,
) -> JsonResult<windmill_common::query_builders::FullDatabaseSchema> {
let pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let (client, connection) = pg.connect().await?;
let join_handle = tokio::spawn(async move { connection.await });
let result = windmill_common::query_builders::pg_get_full_schema(&client)
.await
.map_err(Error::internal_err)?;
drop(client);
join_handle
.await
.map_err(|e| Error::internal_err(format!("join error: {}", e)))?
.map_err(|e| Error::internal_err(format!("tokio_postgres error: {}", e)))?;
Ok(Json(result))
}
async fn edit_ducklake_config(
authed: ApiAuthed,
Extension(db): Extension<DB>,
@@ -3723,7 +3297,7 @@ async fn clone_scripts(
extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,
envs, concurrent_limit, concurrency_time_window_s, cache_ttl,
dedicated_worker, ws_error_handler_muted, priority, timeout,
delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,
delete_after_use, restart_unless_cancelled, concurrency_key,
visible_to_runner_only, auto_kind, codebase, has_preprocessor,
on_behalf_of_email, assets, modules
)
@@ -3733,7 +3307,7 @@ async fn clone_scripts(
extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,
envs, concurrent_limit, concurrency_time_window_s, cache_ttl,
dedicated_worker, ws_error_handler_muted, priority, timeout,
delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,
delete_after_use, restart_unless_cancelled, concurrency_key,
visible_to_runner_only, auto_kind, codebase, has_preprocessor,
on_behalf_of_email, assets, modules
FROM script
@@ -4158,120 +3732,6 @@ async fn create_workspace_fork_branch(
))
}
/// Update a forked workspace's datatable config to point to the new database.
/// For instance datatables: updates resource_path in the datatable config.
/// For resource datatables: updates the resource's dbname and sets ws_specific.
/// Snapshot the schema from the source datatable by connecting to its database.
async fn snapshot_datatable_schema(
db: &DB,
parent_w_id: &str,
dt_name: &str,
) -> Result<serde_json::Value> {
let pg = get_datatable_resource_from_db_unchecked(db, parent_w_id, dt_name).await?;
let pg: PgDatabase = serde_json::from_value(pg)
.map_err(|e| Error::internal_err(format!("Failed to parse db credentials: {}", e)))?;
let (client, connection) = pg.connect().await?;
let join_handle = tokio::spawn(async move { connection.await });
let schema = windmill_common::query_builders::pg_get_full_schema(&client)
.await
.map_err(Error::internal_err)?;
drop(client);
join_handle
.await
.map_err(|e| Error::internal_err(format!("join error: {}", e)))?
.map_err(|e| Error::internal_err(format!("tokio_postgres error: {}", e)))?;
serde_json::to_value(schema)
.map_err(|e| Error::internal_err(format!("Failed to serialize schema: {}", e)))
}
async fn apply_forked_datatable(
db: &DB,
tx: &mut Transaction<'_, Postgres>,
parent_w_id: &str,
forked_w_id: &str,
fdt: &ForkedDatatableInfo,
) -> Result<()> {
windmill_common::validate_dbname(&fdt.new_dbname)?;
if !fdt.new_dbname.starts_with("wm_fork_") {
return Err(Error::BadRequest(format!(
"Forked datatable database name '{}' must start with 'wm_fork_'",
fdt.new_dbname
)));
}
// Snapshot the schema from the source (parent) datatable
let schema = snapshot_datatable_schema(db, parent_w_id, &fdt.name).await?;
let forked_from = serde_json::json!({ "schema": schema });
// Read the datatable config from the forked workspace
let config_val = sqlx::query_scalar!(
"SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
forked_w_id,
&fdt.name
)
.fetch_optional(&mut **tx)
.await?
.flatten()
.ok_or_else(|| {
Error::NotFound(format!(
"Datatable '{}' not found in workspace '{}'",
fdt.name, forked_w_id
))
})?;
let dt: DataTable = serde_json::from_value(config_val)
.map_err(|e| Error::internal_err(format!("Failed to parse datatable config: {}", e)))?;
if dt.database.resource_type == DataTableCatalogResourceType::Instance {
// Instance: update resource_path to the new dbname
sqlx::query!(
r#"UPDATE workspace_settings
SET datatable = jsonb_set(
jsonb_set(datatable, ARRAY['datatables', $2, 'database', 'resource_path'], to_jsonb($3::text)),
ARRAY['datatables', $2, 'forked_from'], $4::jsonb
)
WHERE workspace_id = $1"#,
forked_w_id,
&fdt.name,
&fdt.new_dbname,
forked_from,
)
.execute(&mut **tx)
.await?;
} else {
// Resource: update the resource's dbname and set ws_specific
let resource_path = &dt.database.resource_path;
sqlx::query!(
r#"UPDATE resource
SET value = jsonb_set(value, '{dbname}', to_jsonb($3::text)),
ws_specific = true
WHERE workspace_id = $1 AND path = $2"#,
forked_w_id,
resource_path,
&fdt.new_dbname,
)
.execute(&mut **tx)
.await?;
// Set forked_from on the datatable config
sqlx::query!(
r#"UPDATE workspace_settings
SET datatable = jsonb_set(datatable, ARRAY['datatables', $2, 'forked_from'], $3::jsonb)
WHERE workspace_id = $1"#,
forked_w_id,
&fdt.name,
forked_from,
)
.execute(&mut **tx)
.await?;
}
Ok(())
}
async fn create_workspace_fork(
authed: ApiAuthed,
Extension(db): Extension<DB>,
@@ -4353,11 +3813,6 @@ async fn create_workspace_fork(
// Clone all data from the parent workspace using Rust implementation
clone_workspace_data(&mut tx, &parent_workspace_id, &forked_id).await?;
// Update forked datatable settings to point to new databases
for fdt in &nw.forked_datatables {
apply_forked_datatable(&db, &mut tx, &parent_workspace_id, &forked_id, fdt).await?;
}
audit_log(
&mut *tx,
&authed,
@@ -4369,7 +3824,6 @@ async fn create_workspace_fork(
)
.await?;
tx.commit().await?;
Ok(format!("Created forked workspace {}", &forked_id))
}
@@ -4590,13 +4044,6 @@ async fn invite_user(
) -> Result<(StatusCode, String)> {
require_admin(is_admin, &username)?;
#[cfg(not(feature = "enterprise"))]
if w_id == "admins" {
return Err(Error::BadRequest(
"The admins workspace is reserved for superadmins. Members cannot be added to it without an enterprise license.".to_string(),
));
}
nu.email = nu.email.to_lowercase();
let mut tx = db.begin().await?;
@@ -4662,14 +4109,6 @@ async fn add_user(
Json(mut nu): Json<NewWorkspaceUser>,
) -> Result<(StatusCode, String)> {
require_admin(authed.is_admin, &authed.username)?;
#[cfg(not(feature = "enterprise"))]
if w_id == "admins" {
return Err(Error::BadRequest(
"The admins workspace is reserved for superadmins. Members cannot be added to it without an enterprise license.".to_string(),
));
}
nu.email = nu.email.to_lowercase();
let mut tx = db.begin().await?;
@@ -6081,7 +5520,7 @@ async fn compare_two_resources(
) -> Result<ItemComparison> {
// Get resource from each workspace
let source_resource = sqlx::query!(
"SELECT value, description, resource_type, ws_specific
"SELECT value, description, resource_type
FROM resource
WHERE workspace_id = $1 AND path = $2",
source_workspace_id,
@@ -6091,7 +5530,7 @@ async fn compare_two_resources(
.await?;
let target_resource = sqlx::query!(
"SELECT value, description, resource_type, ws_specific
"SELECT value, description, resource_type
FROM resource
WHERE workspace_id = $1 AND path = $2",
fork_workspace_id,
@@ -6100,17 +5539,6 @@ async fn compare_two_resources(
.fetch_optional(db)
.await?;
// If either side is ws_specific, consider unchanged
let source_ws_specific = source_resource.as_ref().map_or(false, |r| r.ws_specific);
let target_ws_specific = target_resource.as_ref().map_or(false, |r| r.ws_specific);
if source_ws_specific || target_ws_specific {
return Ok(ItemComparison {
has_changes: false,
exists_in_source: source_resource.is_some(),
exists_in_fork: target_resource.is_some(),
});
}
let mut has_changes = false;
// Check metadata differences

View File

@@ -23,10 +23,8 @@ use windmill_common::worker::CLOUD_HOSTED;
use windmill_common::{
auth::is_super_admin_email,
db::UserDB,
error::{Error, Result},
utils::require_admin,
workspaces::DataTable,
};
use windmill_queue::schedule::{get_schedule_opt, push_scheduled_job};
@@ -539,14 +537,6 @@ pub(crate) async fn change_workspace_id(
.execute(&mut *tx)
.await?;
info!("Deleting raw_script_temp table");
sqlx::query!(
"DELETE FROM raw_script_temp WHERE workspace_id = $1",
&old_id
)
.execute(&mut *tx)
.await?;
info!("Updating resource table");
sqlx::query!(
"UPDATE resource SET workspace_id = $1 WHERE workspace_id = $2",
@@ -837,10 +827,6 @@ pub(crate) async fn delete_workspace(
// NATS triggers have on delete cascade
sqlx::query!("DELETE FROM raw_script_temp WHERE workspace_id = $1", &w_id)
.execute(&mut *tx)
.await?;
sqlx::query!("DELETE FROM workspace WHERE id = $1", &w_id)
.execute(&mut *tx)
.await?;
@@ -860,157 +846,6 @@ pub(crate) async fn delete_workspace(
Ok(format!("Deleted workspace {}", &w_id))
}
#[derive(Deserialize)]
pub struct DropForkedDatatableDatabasesRequest {
datatable_names: Vec<String>,
}
/// Drop forked datatable databases. Returns errors per datatable that failed.
/// Same permission as delete_workspace: fork owner or super admin.
pub async fn drop_forked_datatable_databases(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(req): Json<DropForkedDatatableDatabasesRequest>,
) -> Result<Json<Vec<String>>> {
// Same permission check as delete_workspace: fork owner or super admin
let mut tx = db.begin().await?;
if !(w_id.starts_with(WM_FORK_PREFIX) && is_workspace_owner(&authed, &w_id, &mut tx).await?) {
require_super_admin(&db, &authed.email).await?;
}
tx.commit().await?;
let parent_w_id = sqlx::query_scalar!(
"SELECT parent_workspace_id FROM workspace WHERE id = $1",
&w_id
)
.fetch_optional(&db)
.await?
.flatten()
.ok_or_else(|| Error::BadRequest("No parent workspace found".to_string()))?;
let datatable_config = sqlx::query_scalar!(
"SELECT datatable->'datatables' FROM workspace_settings WHERE workspace_id = $1",
&w_id
)
.fetch_optional(&db)
.await?
.flatten()
.unwrap_or(serde_json::json!({}));
let datatables: HashMap<String, DataTable> =
serde_json::from_value(datatable_config).unwrap_or_default();
let mut errors: Vec<String> = Vec::new();
for dt_name in &req.datatable_names {
let dt = match datatables.get(dt_name) {
Some(dt) if dt.forked_from.is_some() => dt,
_ => continue,
};
if dt.database.resource_type
== windmill_common::workspaces::DataTableCatalogResourceType::Instance
{
let db_to_drop = &dt.database.resource_path;
if !db_to_drop.starts_with("wm_fork_") {
errors.push(format!(
"Refusing to drop instance database '{}' for datatable://{}: name does not start with 'wm_fork_'",
db_to_drop, dt_name
));
continue;
}
if let Err(e) = windmill_common::drop_custom_instance_database(&db, db_to_drop).await {
errors.push(format!(
"Could not drop instance database '{}' for datatable://{}: {}",
db_to_drop, dt_name, e
));
}
} else {
let fork_pg = match crate::workspaces::resolve_pg_source_checked(
&db,
&user_db,
&authed,
&w_id,
&format!("datatable://{}", dt_name),
)
.await
{
Ok(pg) => pg,
Err(e) => {
errors.push(format!(
"Could not resolve fork resource for datatable://{}: {}",
dt_name, e
));
continue;
}
};
// We cannot drop the current database, so we connect to the parent's version to run DROP DATABASE on
// the forked version
let parent_pg = match crate::workspaces::resolve_pg_source_checked(
&db,
&user_db,
&authed,
&parent_w_id,
&format!("datatable://{}", dt_name),
)
.await
{
Ok(pg) => pg,
Err(e) => {
errors.push(format!(
"Could not resolve parent resource for datatable://{}: {}",
dt_name, e
));
continue;
}
};
let db_to_drop = &fork_pg.dbname;
if let Err(e) = windmill_common::validate_dbname(db_to_drop) {
errors.push(format!(
"Invalid database name '{}' for datatable://{}: {}",
db_to_drop, dt_name, e
));
continue;
}
if !db_to_drop.starts_with("wm_fork_") {
errors.push(format!(
"Refusing to drop resource database '{}' for datatable://{}: name does not start with 'wm_fork_'",
db_to_drop, dt_name
));
continue;
}
match parent_pg.connect().await {
Ok((client, connection)) => {
let join_handle = tokio::spawn(async move { connection.await });
if let Err(e) = client
.execute(&format!("DROP DATABASE \"{}\"", db_to_drop), &[])
.await
{
errors.push(format!(
"Could not drop database '{}' for datatable://{}: {}",
db_to_drop, dt_name, e
));
}
drop(client);
let _ = join_handle.await;
}
Err(e) => {
errors.push(format!(
"Could not connect to drop database for datatable://{}: {}",
dt_name, e
));
}
}
}
}
Ok(Json(errors))
}
async fn is_workspace_owner(
authed: &ApiAuthed,
w_id: &str,

View File

@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.679.0
version: 1.673.0
title: Windmill API
contact:
@@ -1079,27 +1079,6 @@ paths:
schema:
$ref: "#/components/schemas/CustomInstanceDb"
/settings/drop_custom_instance_pg_database/{name}:
post:
summary: Drops a custom instance database (superadmin only, irreversible)
operationId: dropCustomInstanceDb
tags:
- setting
parameters:
- in: path
name: name
required: true
schema:
type: string
description: The name of the database to drop
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/settings/global/{key}:
get:
summary: get global settings
@@ -1377,152 +1356,6 @@ paths:
schema:
type: string
/settings/object_storage_usage:
get:
summary: get current or last object storage usage computation (null if never run)
operationId: getObjectStorageUsage
tags:
- setting
responses:
"200":
description: current or last storage-usage computation state
content:
application/json:
schema:
nullable: true
type: object
properties:
running:
type: boolean
started_at:
type: string
format: date-time
finished_at:
type: string
format: date-time
nullable: true
current_prefix:
type: string
nullable: true
scanned_objects:
type: integer
format: int64
folders:
type: array
items:
type: object
properties:
prefix:
type: string
size:
type: integer
format: int64
partial:
type: boolean
required:
- prefix
- size
error:
type: string
nullable: true
required:
- running
- started_at
- scanned_objects
- folders
post:
summary: start a background computation of object storage usage by top-level folder
operationId: computeObjectStorageUsage
tags:
- setting
responses:
"202":
description: computation started
content:
text/plain:
schema:
type: string
/settings/run_log_cleanup:
post:
summary: start a manual cleanup of expired logs from object storage
operationId: runLogCleanup
tags:
- setting
responses:
"202":
description: cleanup started
content:
text/plain:
schema:
type: string
/settings/log_cleanup_status:
get:
summary: get status of the manual log cleanup task
operationId: getLogCleanupStatus
tags:
- setting
responses:
"200":
description: current or last log cleanup status (null if never run)
content:
application/json:
schema:
nullable: true
type: object
properties:
running:
type: boolean
started_at:
type: string
format: date-time
finished_at:
type: string
format: date-time
nullable: true
phase:
type: string
total_service:
type: integer
format: int64
processed_service:
type: integer
format: int64
total_jobs:
type: integer
format: int64
processed_jobs:
type: integer
format: int64
s3_deleted:
type: integer
format: int64
orphans_scanned:
type: integer
format: int64
orphans_deleted:
type: integer
format: int64
errors:
type: integer
format: int64
last_error:
type: string
nullable: true
required:
- running
- started_at
- phase
- total_service
- processed_service
- total_jobs
- processed_jobs
- s3_deleted
- orphans_scanned
- orphans_deleted
- errors
/settings/send_stats:
post:
summary: send stats
@@ -1813,129 +1646,6 @@ paths:
schema:
$ref: "#/components/schemas/SecretMigrationReport"
/settings/test_azure_kv_backend:
post:
summary: test Azure Key Vault connection
operationId: testAzureKvBackend
tags:
- setting
requestBody:
description: Azure Key Vault settings to test
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AzureKeyVaultSettings"
responses:
"200":
description: connection successful
content:
text/plain:
schema:
type: string
/settings/migrate_secrets_to_azure_kv:
post:
summary: migrate secrets from database to Azure Key Vault
operationId: migrateSecretsToAzureKv
tags:
- setting
requestBody:
description: Azure Key Vault settings for migration target
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AzureKeyVaultSettings"
responses:
"200":
description: migration report
content:
application/json:
schema:
$ref: "#/components/schemas/SecretMigrationReport"
/settings/migrate_secrets_from_azure_kv:
post:
summary: migrate secrets from Azure Key Vault to database
operationId: migrateSecretsFromAzureKv
tags:
- setting
requestBody:
description: Azure Key Vault settings for migration source
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AzureKeyVaultSettings"
responses:
"200":
description: migration report
content:
application/json:
schema:
$ref: "#/components/schemas/SecretMigrationReport"
/settings/test_aws_sm_backend:
post:
summary: test connection to AWS Secrets Manager
operationId: testAwsSmBackend
tags:
- setting
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AwsSecretsManagerSettings"
responses:
"200":
description: connection test result
content:
text/plain:
schema:
type: string
/settings/migrate_secrets_to_aws_sm:
post:
summary: migrate secrets from database to AWS Secrets Manager
operationId: migrateSecretsToAwsSm
tags:
- setting
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AwsSecretsManagerSettings"
responses:
"200":
description: migration report
content:
application/json:
schema:
$ref: "#/components/schemas/SecretMigrationReport"
/settings/migrate_secrets_from_aws_sm:
post:
summary: migrate secrets from AWS Secrets Manager to database
operationId: migrateSecretsFromAwsSm
tags:
- setting
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AwsSecretsManagerSettings"
responses:
"200":
description: migration report
content:
application/json:
schema:
$ref: "#/components/schemas/SecretMigrationReport"
/users/email:
get:
summary: get current user email (if logged in)
@@ -3776,16 +3486,7 @@ paths:
schema:
type: array
items:
type: object
required: [name, resource_type, resource_path]
properties:
name:
type: string
resource_type:
type: string
enum: [postgres, instance]
resource_path:
type: string
type: string
/w/{workspace}/workspaces/list_datatable_schemas:
get:
@@ -3858,211 +3559,6 @@ paths:
application/json:
schema: {}
/w/{workspace}/workspaces/create_pg_database:
post:
summary: create a new PostgreSQL database for a datatable
operationId: createPgDatabase
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
description: Create pg database request
required: true
content:
application/json:
schema:
type: object
required: [source, target_dbname]
properties:
source:
type: string
description: "Datatable source to determine connection info: 'datatable://name' or '$res:path'"
target_dbname:
type: string
description: "Name for the new database"
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/drop_forked_datatable_databases:
post:
summary: drop forked datatable databases
operationId: dropForkedDatatableDatabases
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [datatable_names]
properties:
datatable_names:
type: array
items:
type: string
responses:
"200":
description: list of errors (empty if all succeeded)
content:
application/json:
schema:
type: array
items:
type: string
/w/{workspace}/workspaces/import_pg_database:
post:
summary: import a PostgreSQL database from source to target via pg_dump
operationId: importPgDatabase
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
description: Import pg database request
required: true
content:
application/json:
schema:
type: object
required: [source, target, fork_behavior]
properties:
source:
type: string
description: "Source database: 'datatable://name' or '$res:path'"
target:
type: string
description: "Target database: 'datatable://name' or '$res:path'"
target_dbname_override:
type: string
description: "Override the target database name"
fork_behavior:
type: string
enum:
- schema_only
- schema_and_data
- keep_original
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/export_pg_schema:
post:
summary: export the schema of a PostgreSQL database
operationId: exportPgSchema
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
description: Export pg schema request
required: true
content:
application/json:
schema:
type: object
required: [source]
properties:
source:
type: string
description: "Source database: 'datatable://name' or '$res:path'"
responses:
"200":
description: schema dump
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/get_datatable_full_schema:
post:
summary: get the full schema of a datatable database as TableEditorValues
operationId: getDatatableFullSchema
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [source]
properties:
source:
type: string
description: "Source datatable, e.g. 'datatable://main'"
responses:
"200":
description: "Schema as { schema_name: { table_name: TableEditorValues } }"
content:
application/json:
schema:
type: object
additionalProperties:
type: object
additionalProperties:
type: object
required: [name, columns, foreign_keys]
properties:
name:
type: string
columns:
type: array
items:
type: object
required: [name, datatype]
properties:
name:
type: string
datatype:
type: string
primary_key:
type: boolean
default_value:
type: string
nullable:
type: boolean
foreign_keys:
type: array
items:
type: object
required: [columns, on_delete, on_update]
properties:
target_table:
type: string
columns:
type: array
items:
type: object
properties:
source_column:
type: string
target_column:
type: string
on_delete:
type: string
on_update:
type: string
fk_constraint_name:
type: string
pk_constraint_name:
type: string
/w/{workspace}/workspaces/git_sync_enabled:
get:
summary: Check if git sync is available for this workspace
@@ -9072,35 +8568,6 @@ paths:
items:
$ref: "#/components/schemas/FlowConversationMessage"
/w/{workspace}/path_autocomplete/list_paths:
get:
summary: list all paths in a workspace for client-side autocomplete
description: |
Returns the flat list of all item paths visible to the caller across
scripts, flows, apps, raw apps, variables, and resources. Intended to
feed an entirely client-side path autocomplete UI: the frontend fetches
once (server caches per workspace for 60s) and performs all prefix/segment
computation locally. Capped at 20,000 paths (5,000 per table).
operationId: listPathAutocompletePaths
tags:
- path_autocomplete
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: deduplicated path list, sorted lexicographically
content:
application/json:
schema:
type: object
properties:
paths:
type: array
items:
type: string
required:
- paths
/w/{workspace}/raw_apps/list:
get:
summary: list all raw apps
@@ -11266,23 +10733,6 @@ paths:
schema:
type: string
/w/{workspace}/jobs_u/get_flow_all_logs/{id}:
get:
summary: get all logs for a flow job
operationId: getFlowAllLogs
tags:
- job
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- $ref: "#/components/parameters/JobId"
responses:
"200":
description: concatenated logs of all flow steps
content:
text/plain:
schema:
type: string
/w/{workspace}/jobs_u/get_completed_logs_tail/{id}:
get:
summary: get completed job logs tail
@@ -18607,7 +18057,7 @@ paths:
type: object
description: The columns used (for tables)
additionalProperties:
$ref: "#/components/schemas/AssetUsageAccessType"
$ref: "#/components/schemas/AssetUsageAccessType"
created_at:
type: string
format: date-time
@@ -18711,6 +18161,7 @@ paths:
type: string
description: The asset path
/w/{workspace}/volumes/list:
get:
summary: List all volumes in the workspace
@@ -19693,50 +19144,6 @@ components:
type: string
description: Static Vault token for testing/development (optional, if provided this is used instead of JWT authentication)
AzureKeyVaultSettings:
type: object
required:
- vault_url
- tenant_id
- client_id
properties:
vault_url:
type: string
description: Azure Key Vault URL (e.g., https://myvault.vault.azure.net)
tenant_id:
type: string
description: Azure AD tenant ID
client_id:
type: string
description: Azure AD application (client) ID
client_secret:
type: string
description: Azure AD client secret
token:
type: string
description: Static Bearer token for testing/development (optional, if provided this is used instead of OAuth2 authentication)
AwsSecretsManagerSettings:
type: object
required:
- region
properties:
region:
type: string
description: AWS region (e.g., us-east-1)
access_key_id:
type: string
description: AWS Access Key ID (optional, uses default credential chain if not provided)
secret_access_key:
type: string
description: AWS Secret Access Key (optional)
endpoint_url:
type: string
description: Custom endpoint URL for testing (e.g., LocalStack)
prefix:
type: string
description: Prefix for secret names (e.g., windmill/)
SecretMigrationFailure:
type: object
required:
@@ -20165,9 +19572,8 @@ components:
type: boolean
timeout:
type: integer
delete_after_secs:
type: integer
description: If set, delete the job's args, result and logs after this many seconds following job completion
delete_after_use:
type: boolean
visible_to_runner_only:
type: boolean
auto_kind:
@@ -20258,9 +19664,8 @@ components:
type: boolean
timeout:
type: integer
delete_after_secs:
type: integer
description: If set, delete the job's args, result and logs after this many seconds following job completion
delete_after_use:
type: boolean
deployment_message:
type: string
concurrency_key:
@@ -21659,9 +21064,6 @@ components:
resource_type:
type: string
description: The new resource_type to be associated with the resource
ws_specific:
type: boolean
description: When true, the resource is excluded from workspace diff comparisons
labels:
type: array
items:
@@ -24459,18 +23861,6 @@ components:
type: string
color:
type: string
forked_datatables:
type: array
items:
type: object
required: [name, new_dbname]
properties:
name:
type: string
description: "Datatable name"
new_dbname:
type: string
description: "New database name for the fork"
required:
- id
- name
@@ -25122,14 +24512,7 @@ components:
type: string
required:
- resource_type
forked_from:
type: object
description: Fork origin info with schema snapshot
properties:
schema:
type: object
description: Schema snapshot at fork time
additionalProperties: true
DataTableSchema:
type: object
required: [datatable_name, schemas]
@@ -25711,15 +25094,7 @@ components:
kind:
type: string
enum:
[
"script",
"flow",
"app",
"raw_app",
"resource",
"variable",
"resource_type",
]
["script", "flow", "app", "raw_app", "resource", "variable", "resource_type"]
description: Type of the item
path:
type: string

View File

@@ -20,10 +20,9 @@ use windmill_common::ai_cache::current_instance_ai_config_revision;
use windmill_common::ai_providers::{
empty_string_as_none, AIPlatform, AIProvider, ProviderConfig, ProviderModel,
};
use windmill_common::db::UserDB;
use windmill_common::error::{to_anyhow, Error, Result};
use windmill_common::utils::configure_client;
use windmill_common::variables::{get_variable_or_self, get_variable_or_self_as};
use windmill_common::variables::get_variable_or_self;
// AI timeout configuration constants
const AI_TIMEOUT_MIN_SECS: u64 = 1;
@@ -212,35 +211,13 @@ struct AIRequestConfig {
pub custom_headers: HashMap<String, String>,
}
/// Resolve a `$var:` reference. When `user_db`/`authed` are provided the query
/// goes through an RLS-scoped connection so the caller can only read variables
/// they are authorised to access. Without auth context the raw pool is used
/// (appropriate for admin/system paths where the resource was already validated).
async fn resolve_var(
path: String,
db: &DB,
w_id: &str,
user_db: Option<&UserDB>,
authed: Option<&ApiAuthed>,
) -> Result<String> {
match (user_db, authed) {
(Some(udb), Some(auth)) => Ok(get_variable_or_self_as(path, db, udb, auth, w_id).await?),
_ => Ok(get_variable_or_self(path, db, w_id).await?),
}
}
impl AIRequestConfig {
pub async fn new(
provider: &AIProvider,
db: &DB,
w_id: &str,
resource: AIResource,
authed: Option<&ApiAuthed>,
) -> Result<Self> {
// When authed is provided, resolve $var: references through RLS so that
// users can only read variables they have permission to access.
let user_db = authed.map(|_| UserDB::new(db.clone()));
let (
api_key,
access_token,
@@ -267,29 +244,28 @@ impl AIRequestConfig {
provider.get_base_url(resource.base_url, db).await?
};
let api_key = if let Some(api_key) = resource.api_key {
Some(resolve_var(api_key, db, w_id, user_db.as_ref(), authed).await?)
Some(get_variable_or_self(api_key, db, w_id).await?)
} else {
None
};
let organization_id = if let Some(organization_id) = resource.organization_id {
Some(resolve_var(organization_id, db, w_id, user_db.as_ref(), authed).await?)
Some(get_variable_or_self(organization_id, db, w_id).await?)
} else {
None
};
let aws_access_key_id = if let Some(access_key_id) = resource.aws_access_key_id {
Some(resolve_var(access_key_id, db, w_id, user_db.as_ref(), authed).await?)
} else {
None
};
let aws_secret_access_key = if let Some(secret_access_key) =
resource.aws_secret_access_key
{
Some(resolve_var(secret_access_key, db, w_id, user_db.as_ref(), authed).await?)
Some(get_variable_or_self(access_key_id, db, w_id).await?)
} else {
None
};
let aws_secret_access_key =
if let Some(secret_access_key) = resource.aws_secret_access_key {
Some(get_variable_or_self(secret_access_key, db, w_id).await?)
} else {
None
};
let aws_session_token = if let Some(session_token) = resource.aws_session_token {
Some(resolve_var(session_token, db, w_id, user_db.as_ref(), authed).await?)
Some(get_variable_or_self(session_token, db, w_id).await?)
} else {
None
};
@@ -311,13 +287,11 @@ impl AIRequestConfig {
}
AIResource::OAuth(resource) => {
let user = if let Some(user) = resource.user.clone() {
Some(resolve_var(user, db, w_id, user_db.as_ref(), authed).await?)
Some(get_variable_or_self(user, db, w_id).await?)
} else {
None
};
let token =
Self::get_token_using_oauth(resource, db, w_id, user_db.as_ref(), authed)
.await?;
let token = Self::get_token_using_oauth(resource, db, w_id).await?;
let base_url = provider.get_base_url(None, db).await?;
(
@@ -357,13 +331,10 @@ impl AIRequestConfig {
mut resource: AIOAuthResource,
db: &DB,
w_id: &str,
user_db: Option<&UserDB>,
authed: Option<&ApiAuthed>,
) -> Result<String> {
resource.client_id = resolve_var(resource.client_id, db, w_id, user_db, authed).await?;
resource.client_secret =
resolve_var(resource.client_secret, db, w_id, user_db, authed).await?;
resource.token_url = resolve_var(resource.token_url, db, w_id, user_db, authed).await?;
resource.client_id = get_variable_or_self(resource.client_id, db, w_id).await?;
resource.client_secret = get_variable_or_self(resource.client_secret, db, w_id).await?;
resource.token_url = get_variable_or_self(resource.token_url, db, w_id).await?;
let mut params = HashMap::new();
params.insert("grant_type", "client_credentials");
params.insert("scope", "https://cognitiveservices.azure.com/.default");
@@ -807,7 +778,6 @@ async fn proxy(
let forced_resource_path = headers
.get("X-Resource-Path")
.map(|v| v.to_str().unwrap_or("").to_string());
let is_user_specified_resource = forced_resource_path.is_some();
let request_config = match workspace_cache {
Some(request_cache) if !request_cache.is_expired() && forced_resource_path.is_none() => {
request_cache.config
@@ -891,22 +861,8 @@ async fn proxy(
let resource = serde_json::from_str::<AIResource>(resource.0.get())
.map_err(|e| Error::BadRequest(e.to_string()))?;
// Enforce RLS on $var: resolution when the resource path was
// user-specified (X-Resource-Path header) so users can only read
// variables they have permission to access.
let enforce_authed = if is_user_specified_resource {
Some(&authed)
} else {
None
};
let request_config = AIRequestConfig::new(
&provider,
&db,
&resource_workspace,
resource,
enforce_authed,
)
.await?;
let request_config =
AIRequestConfig::new(&provider, &db, &resource_workspace, resource).await?;
if save_to_cache {
AI_REQUEST_CACHE.insert(
(w_id.clone(), provider.clone()),

View File

@@ -5,9 +5,8 @@ use axum::{
http::HeaderMap,
response::{IntoResponse, Response},
};
use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _};
use bytes::Bytes;
use http::header::CONTENT_TYPE;
use http::{header::CONTENT_TYPE, StatusCode};
use serde::{Deserialize, Serialize};
use serde_json::value::RawValue;
use sqlx::types::JsonRawValue;
@@ -32,7 +31,6 @@ pub enum RawBody {
Xml(String),
UrlEncoded(Bytes),
Multipart(Multipart),
RawBytes(Bytes),
Empty,
}
@@ -187,16 +185,6 @@ impl RawWebhookArgs {
body: Body::HashMap(HashMap::new()),
metadata: WebhookArgsMetadata { raw_string: Some(s), ..self.metadata },
}),
RawBody::RawBytes(bytes) => {
let s = match String::from_utf8(bytes.to_vec()) {
Ok(s) => s,
Err(e) => BASE64_STANDARD.encode(e.into_bytes()),
};
Ok(WebhookArgs {
body: Body::HashMap(HashMap::new()),
metadata: WebhookArgsMetadata { raw_string: Some(s), ..self.metadata },
})
}
RawBody::UrlEncoded(bytes) => {
let mut metadata = self.metadata;
if use_raw {
@@ -459,10 +447,7 @@ where
Ok(RawWebhookArgs { body: RawBody::Multipart(multipart), metadata })
} else {
let bytes = Bytes::from_request(request, _state)
.await
.map_err(IntoResponse::into_response)?;
Ok(RawWebhookArgs { body: RawBody::RawBytes(bytes), metadata })
Err(StatusCode::UNSUPPORTED_MEDIA_TYPE.into_response())
}
}

View File

@@ -6,12 +6,7 @@
* LICENSE-AGPL for a copy of the license.
*/
use axum::{
extract::Query,
http::StatusCode,
routing::{get, post},
Extension, Json, Router,
};
use axum::{extract::Query, routing::get, Extension, Json, Router};
use serde::{Deserialize, Serialize};
use windmill_common::error::JsonResult;
@@ -20,11 +15,7 @@ use crate::db::{ApiAuthed, DB};
use crate::utils::require_super_admin;
pub fn global_service() -> Router {
Router::new()
.route("/", get(get_db_health))
.route("/jobs", get(get_db_health_jobs))
.route("/slow_queries", get(get_slow_queries))
.route("/slow_queries/reset", post(reset_slow_queries))
Router::new().route("/", get(get_db_health))
}
// --- Response types ---
@@ -40,18 +31,14 @@ pub enum HealthLevel {
#[derive(Serialize)]
pub struct DbHealthResponse {
pub database_size: DatabaseSizeInfo,
pub job_retention: JobRetentionInfo,
pub large_results: LargeResultsInfo,
pub connection_pool: ConnectionPoolInfo,
pub table_maintenance: Vec<TableMaintenanceInfo>,
pub slow_queries: Option<SlowQueriesInfo>,
pub datatables: Vec<DatatableInfo>,
}
#[derive(Serialize)]
pub struct DbHealthJobsResponse {
pub job_retention: JobRetentionInfo,
pub large_results: LargeResultsInfo,
}
#[derive(Serialize)]
pub struct DatabaseSizeInfo {
pub total_size_bytes: i64,
@@ -115,8 +102,6 @@ pub struct TableMaintenanceInfo {
pub struct SlowQueriesInfo {
pub queries: Vec<SlowQueryRow>,
pub message: Option<String>,
/// When stats were last reset (from pg_stat_statements_info.stats_reset, PG 14+)
pub stats_reset: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Serialize)]
@@ -145,45 +130,37 @@ struct DbHealthQuery {
scan_limit: Option<i64>,
}
#[derive(Deserialize, Clone, Copy)]
#[serde(rename_all = "snake_case")]
enum SlowQuerySort {
Total,
Mean,
Calls,
}
impl SlowQuerySort {
fn order_by(&self) -> &'static str {
match self {
SlowQuerySort::Total => "total_exec_time DESC",
SlowQuerySort::Mean => "mean_exec_time DESC",
SlowQuerySort::Calls => "calls DESC",
}
}
}
#[derive(Deserialize)]
struct SlowQueriesQuery {
sort: Option<SlowQuerySort>,
}
async fn get_db_health(
ApiAuthed { email, .. }: ApiAuthed,
Extension(db): Extension<DB>,
Query(query): Query<DbHealthQuery>,
) -> JsonResult<DbHealthResponse> {
require_super_admin(&db, &email).await?;
let (database_size, connection_pool, table_maintenance, slow_queries, datatables) = tokio::try_join!(
let scan_limit = query.scan_limit.unwrap_or(10_000).clamp(1_000, 1_000_000);
let (
database_size,
job_retention,
large_results,
connection_pool,
table_maintenance,
slow_queries,
datatables,
) = tokio::try_join!(
fetch_database_size(&db),
fetch_job_retention(&db),
fetch_large_results(&db, scan_limit),
fetch_connection_pool(&db),
fetch_table_maintenance(&db),
fetch_slow_queries(&db, SlowQuerySort::Total),
fetch_slow_queries(&db),
fetch_datatables(&db),
)?;
Ok(Json(DbHealthResponse {
database_size,
job_retention,
large_results,
connection_pool,
table_maintenance,
slow_queries,
@@ -191,49 +168,6 @@ async fn get_db_health(
}))
}
async fn get_db_health_jobs(
ApiAuthed { email, .. }: ApiAuthed,
Extension(db): Extension<DB>,
Query(query): Query<DbHealthQuery>,
) -> JsonResult<DbHealthJobsResponse> {
require_super_admin(&db, &email).await?;
let scan_limit = query.scan_limit.unwrap_or(10_000).clamp(1_000, 1_000_000);
let (job_retention, large_results) = tokio::try_join!(
fetch_job_retention(&db),
fetch_large_results(&db, scan_limit),
)?;
Ok(Json(DbHealthJobsResponse { job_retention, large_results }))
}
async fn get_slow_queries(
ApiAuthed { email, .. }: ApiAuthed,
Extension(db): Extension<DB>,
Query(query): Query<SlowQueriesQuery>,
) -> JsonResult<Option<SlowQueriesInfo>> {
require_super_admin(&db, &email).await?;
let sort = query.sort.unwrap_or(SlowQuerySort::Total);
Ok(Json(fetch_slow_queries(&db, sort).await?))
}
async fn reset_slow_queries(
ApiAuthed { email, .. }: ApiAuthed,
Extension(db): Extension<DB>,
) -> windmill_common::error::Result<StatusCode> {
require_super_admin(&db, &email).await?;
sqlx::query("SELECT pg_stat_statements_reset()")
.execute(&db)
.await
.map_err(|e| {
windmill_common::error::Error::InternalErr(format!(
"Failed to reset pg_stat_statements (is the extension enabled?): {e}"
))
})?;
Ok(StatusCode::NO_CONTENT)
}
// --- Diagnostic queries ---
async fn fetch_database_size(db: &DB) -> windmill_common::error::Result<DatabaseSizeInfo> {
@@ -473,7 +407,6 @@ async fn fetch_table_maintenance(
LEFT JOIN pg_class p ON p.oid = i.inhparent
) sub
GROUP BY table_name
HAVING SUM(live_tuples) + SUM(dead_tuples) >= 1000
ORDER BY SUM(dead_tuples) DESC"#
)
.fetch_all(db)
@@ -508,10 +441,7 @@ async fn fetch_table_maintenance(
.collect())
}
async fn fetch_slow_queries(
db: &DB,
sort: SlowQuerySort,
) -> windmill_common::error::Result<Option<SlowQueriesInfo>> {
async fn fetch_slow_queries(db: &DB) -> windmill_common::error::Result<Option<SlowQueriesInfo>> {
let ext_exists: bool = sqlx::query_scalar!(
r#"SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements') as "exists!""#
)
@@ -525,54 +455,35 @@ async fn fetch_slow_queries(
"pg_stat_statements extension is not installed. Enable it for slow query insights."
.to_string(),
),
stats_reset: None,
}));
}
// Use raw query since pg_stat_statements may not exist at compile time.
// ORDER BY column is controlled via an enum (SlowQuerySort) so only safe
// whitelisted column names reach the query — no SQL injection risk.
let query = format!(
// Use raw query since pg_stat_statements may not exist at compile time
let rows: Vec<SlowQueryRow> = sqlx::query_as::<_, (String, i64, f64, f64)>(
r#"SELECT
LEFT(query, 500),
LEFT(query, 200),
calls::bigint,
total_exec_time::float8,
mean_exec_time::float8
FROM pg_stat_statements
WHERE query NOT LIKE '%pg_stat_statements%'
ORDER BY {}
LIMIT 50"#,
sort.order_by()
);
let rows: Vec<SlowQueryRow> = sqlx::query_as::<_, (String, i64, f64, f64)>(&query)
.fetch_all(db)
.await?
.into_iter()
.map(
|(query, calls, total_exec_time_ms, mean_exec_time_ms)| SlowQueryRow {
query,
calls,
total_exec_time_ms,
mean_exec_time_ms,
},
)
.collect();
ORDER BY mean_exec_time DESC
LIMIT 10"#,
)
.fetch_all(db)
.await?
.into_iter()
.map(
|(query, calls, total_exec_time_ms, mean_exec_time_ms)| SlowQueryRow {
query,
calls,
total_exec_time_ms,
mean_exec_time_ms,
},
)
.collect();
// pg_stat_statements_info exists in PG 14+; tolerate its absence
let stats_reset: Option<chrono::DateTime<chrono::Utc>> =
sqlx::query_scalar::<_, Option<chrono::DateTime<chrono::Utc>>>(
"SELECT stats_reset FROM pg_stat_statements_info",
)
.fetch_one(db)
.await
.ok()
.flatten();
Ok(Some(SlowQueriesInfo {
queries: rows,
message: None,
stats_reset,
}))
Ok(Some(SlowQueriesInfo { queries: rows, message: None }))
}
async fn fetch_datatables(db: &DB) -> windmill_common::error::Result<Vec<DatatableInfo>> {

View File

@@ -378,7 +378,6 @@ pub fn workspace_unauthed_service() -> Router {
.route("/get_root_job_id/{id}", get(get_root_job))
.route("/get/{id}", get(get_job))
.route("/get_logs/{id}", get(get_job_logs))
.route("/get_flow_all_logs/{id}", get(get_flow_all_logs))
.route(
"/get_completed_logs_tail/{id}",
get(get_completed_job_logs_tail),
@@ -1687,217 +1686,6 @@ async fn get_job_logs(
}
}
async fn resolve_logs_to_string(
log_offset: i32,
logs: &str,
log_file_index: &Option<Vec<String>>,
) -> String {
use futures::StreamExt;
#[cfg(all(feature = "enterprise", feature = "parquet"))]
if let Some(stream) =
windmill_object_store::get_logs_from_store(log_offset, logs, log_file_index).await
{
let mut result = String::new();
futures::pin_mut!(stream);
while let Some(Ok(bytes)) = stream.next().await {
result.push_str(&String::from_utf8_lossy(&bytes));
}
return result;
}
if let Some(stream) =
windmill_common::jobs::get_logs_from_disk(log_offset, logs, log_file_index).await
{
let mut result = String::new();
futures::pin_mut!(stream);
while let Some(Ok(bytes)) = stream.next().await {
result.push_str(&String::from_utf8_lossy(&bytes));
}
return result;
}
logs.to_string()
}
async fn get_flow_all_logs(
OptAuthed(opt_authed): OptAuthed,
opt_tokened: OptTokened,
Extension(db): Extension<DB>,
Path((w_id, id)): Path<(String, Uuid)>,
) -> error::Result<Response> {
let tags = opt_authed
.as_ref()
.map(|authed| get_scope_tags(authed).map(|v| v.iter().map(|s| s.to_string()).collect_vec()))
.flatten();
// Verify the root job exists and check auth
let root_job = sqlx::query!(
"SELECT created_by FROM v2_job WHERE id = $1 AND workspace_id = $2 AND ($3::text[] IS NULL OR tag = ANY($3))",
id,
w_id,
tags.as_ref().map(|v| v.as_slice())
)
.fetch_optional(&db)
.await?;
let root_job = not_found_if_none(root_job, "Job", id.to_string())?;
if opt_authed.is_none() && root_job.created_by != "anonymous" {
return Err(Error::BadRequest(
"As a non logged in user, you can only see jobs ran by anonymous users".to_string(),
));
}
log_job_view(
&db,
opt_authed.as_ref(),
opt_tokened.token.as_deref(),
&w_id,
&id,
)
.await?;
// Fetch all jobs in the flow tree using recursive CTE.
// Uses a materialized id_path for depth-first ordering so children
// appear right after their parent (e.g. iteration 1 → its steps → iteration 2 → ...).
// Extracts parent_module_type (branchall, forloopflow, etc.) from parent's flow definition.
let records = sqlx::query!(
"WITH RECURSIVE job_tree AS (
SELECT j.id, j.kind::text, j.flow_step_id, j.parent_job,
'' as path_label, 0 as depth,
j.id::text as id_path,
''::text as parent_module_type
FROM v2_job j
WHERE j.id = $2 AND j.workspace_id = $1
UNION ALL
SELECT j.id, j.kind::text, j.flow_step_id, j.parent_job,
CASE
WHEN jt.path_label = '' THEN COALESCE(j.flow_step_id, '')
ELSE jt.path_label || '/' || COALESCE(j.flow_step_id, '')
END,
jt.depth + 1,
jt.id_path || '/' || j.id::text,
COALESCE((
SELECT m->'value'->>'type'
FROM v2_job parent_j
LEFT JOIN flow f ON f.path = parent_j.runnable_path
AND f.workspace_id = parent_j.workspace_id
LEFT JOIN flow_node fn ON fn.id = parent_j.runnable_id
CROSS JOIN LATERAL jsonb_array_elements(
COALESCE(parent_j.raw_flow, f.value, fn.flow)->'modules'
) m
WHERE parent_j.id = jt.id
AND m->>'id' = j.flow_step_id
LIMIT 1
), '')::text
FROM v2_job j
JOIN job_tree jt ON j.parent_job = jt.id
WHERE j.workspace_id = $1
),
with_sibling_index AS (
SELECT jt.*,
ROW_NUMBER() OVER (
PARTITION BY jt.parent_job, jt.flow_step_id
ORDER BY jt.id
) as sibling_index,
COUNT(*) OVER (
PARTITION BY jt.parent_job, jt.flow_step_id
) as sibling_count
FROM job_tree jt
)
SELECT w.id, w.kind, w.flow_step_id, w.path_label,
w.sibling_index::int as sibling_index,
w.sibling_count::int as sibling_count,
w.depth::int as depth,
w.parent_module_type,
coalesce(job_logs.logs, '') as logs,
COALESCE(job_logs.log_offset, 0) as log_offset,
job_logs.log_file_index
FROM with_sibling_index w
LEFT JOIN job_logs ON job_logs.job_id = w.id
ORDER BY w.id_path ASC",
w_id,
id,
)
.fetch_all(&db)
.await?;
let mut all_logs = String::new();
for record in &records {
let kind = record.kind.as_deref().unwrap_or("");
let step_id = record.flow_step_id.as_deref().unwrap_or("");
let depth = record.depth.unwrap_or(0);
let sibling_count = record.sibling_count.unwrap_or(1);
let sibling_index = record.sibling_index.unwrap_or(0);
let parent_module_type = record.parent_module_type.as_deref().unwrap_or("");
// Build a descriptive label
let path = record.path_label.as_deref().unwrap_or(step_id);
let kind_label = match parent_module_type {
"branchall" => " branchall",
"branchone" => " branchone",
"forloopflow" => " forloop",
"whileloopflow" => " whileloop",
"aiagent" => " ai-agent",
_ => "",
};
let label = if depth == 0 {
"Flow".to_string()
} else if matches!(
kind,
"flow" | "flowpreview" | "flownode" | "singlestepflow" | "aiagent"
) {
// Intermediate flow job (loop iteration or branch)
if parent_module_type == "branchone" {
let branch_label = if sibling_index == 1 {
"default".to_string()
} else {
format!("{}", sibling_index - 1)
};
format!("Step {}{} (branch {})", path, kind_label, branch_label)
} else if parent_module_type == "branchall" {
format!("Step {}{} (branch {})", path, kind_label, sibling_index)
} else if parent_module_type == "forloopflow" || parent_module_type == "whileloopflow" {
format!(
"Step {}{} (iteration {}/{})",
path, kind_label, sibling_index, sibling_count
)
} else if sibling_count > 1 {
format!(
"Step {} (iteration {}/{})",
path, sibling_index, sibling_count
)
} else {
format!("Step {} (subflow)", path)
}
} else if sibling_count > 1 {
// Simple module optimization: forloop/whileloop with single step
// runs iterations as direct script jobs instead of subflows
format!(
"Step {}{} (iteration {}/{})",
path, kind_label, sibling_index, sibling_count
)
} else {
format!("Step {}", path)
};
let job_id = record.id.map(|u| u.to_string()).unwrap_or_default();
all_logs.push_str(&format!("\n=== {} (Job: {}) ===\n", label, job_id));
let logs = record.logs.as_deref().unwrap_or("");
let resolved =
resolve_logs_to_string(record.log_offset.unwrap_or(0), logs, &record.log_file_index)
.await;
all_logs.push_str(&resolved);
all_logs.push('\n');
}
Ok(content_plain(Body::from(all_logs)))
}
async fn get_args(
OptAuthed(opt_authed): OptAuthed,
opt_tokened: OptTokened,
@@ -2221,7 +2009,7 @@ async fn count_completed_jobs_detail(
sqlb.join("v2_job USING (id)");
sqlb.field("COUNT(*) as count");
if !(w_id == "admins" && query.all_workspaces.unwrap_or(false)) {
if !query.all_workspaces.unwrap_or(false) {
sqlb.and_where_eq("v2_job.workspace_id", "?".bind(&w_id));
}
@@ -4281,51 +4069,49 @@ pub async fn run_workflow_as_code(
)
.await?;
let (job_payload, tag, _delete_after_use, _delete_after_secs, timeout, on_behalf_of) =
match job.job_kind {
JobKind::Preview => (
JobPayload::Code(RawCode {
hash: None,
content: raw_code.unwrap_or_default(),
path: job.script_path,
language: job.language.unwrap_or_else(|| ScriptLang::Deno),
lock: raw_lock,
concurrency_settings: concurrency_settings
.maybe_fallback(
windmill_queue::custom_concurrency_key(&db, &job.id)
.await
.map_err(to_anyhow)?,
job.concurrent_limit,
job.concurrency_time_window_s,
)
.into(),
cache_ttl: job.cache_ttl,
cache_ignore_s3_path: job.cache_ignore_s3_path,
dedicated_worker: None,
// TODO(debouncing): enable for this mode
debouncing_settings: DebouncingSettings::default(),
modules: None,
}),
Some(job.tag.clone()),
None,
None,
run_query.timeout,
None,
),
JobKind::Script => {
let userdb_authed =
UserDbWithAuthed { db: user_db.clone(), authed: &authed.to_authed_ref() };
script_path_to_payload(
job.script_path(),
Some(userdb_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?
}
_ => return Err(anyhow::anyhow!("Not supported").into()),
};
let (job_payload, tag, _delete_after_use, timeout, on_behalf_of) = match job.job_kind {
JobKind::Preview => (
JobPayload::Code(RawCode {
hash: None,
content: raw_code.unwrap_or_default(),
path: job.script_path,
language: job.language.unwrap_or_else(|| ScriptLang::Deno),
lock: raw_lock,
concurrency_settings: concurrency_settings
.maybe_fallback(
windmill_queue::custom_concurrency_key(&db, &job.id)
.await
.map_err(to_anyhow)?,
job.concurrent_limit,
job.concurrency_time_window_s,
)
.into(),
cache_ttl: job.cache_ttl,
cache_ignore_s3_path: job.cache_ignore_s3_path,
dedicated_worker: None,
// TODO(debouncing): enable for this mode
debouncing_settings: DebouncingSettings::default(),
modules: None,
}),
Some(job.tag.clone()),
None,
run_query.timeout,
None,
),
JobKind::Script => {
let userdb_authed =
UserDbWithAuthed { db: user_db.clone(), authed: &authed.to_authed_ref() };
script_path_to_payload(
job.script_path(),
Some(userdb_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?
}
_ => return Err(anyhow::anyhow!("Not supported").into()),
};
if *CLOUD_HOSTED {
tracing::info!("workflow_as_code_tracing id {i} ");
@@ -4556,15 +4342,14 @@ pub async fn run_wait_result_job_by_path_get(
let user_db_with_authed =
UserDbWithAuthed { db: user_db.clone(), authed: &authed.to_authed_ref() };
let (job_payload, tag, delete_after_use, delete_after_secs, timeout, on_behalf_authed) =
script_path_to_payload(
script_path,
Some(user_db_with_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?;
let (job_payload, tag, delete_after_use, timeout, on_behalf_authed) = script_path_to_payload(
script_path,
Some(user_db_with_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?;
let tag = run_query.tag.clone().or(tag);
check_tag_available_for_workspace(&db, &w_id, &tag, &authed).await?;
@@ -4620,7 +4405,9 @@ pub async fn run_wait_result_job_by_path_get(
tx.commit().await?;
let wait_result = run_wait_result(&db, uuid, &w_id, None, &authed.username).await;
handle_delete_after_completion(&db, uuid, &w_id, delete_after_use, delete_after_secs).await?;
if delete_after_use.unwrap_or(false) {
delete_job_metadata_after_use(&db, uuid).await?;
}
return wait_result;
}
@@ -4700,15 +4487,14 @@ pub async fn run_wait_result_script_by_path_internal(
check_queue_too_long(&db, QUEUE_LIMIT_WAIT_RESULT.or(run_query.queue_limit)).await?;
let db_authed = UserDbWithAuthed { db: user_db.clone(), authed: &authed.to_authed_ref() };
let (job_payload, tag, delete_after_use, delete_after_secs, timeout, on_behalf_of) =
script_path_to_payload(
script_path.to_path(),
Some(db_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?;
let (job_payload, tag, delete_after_use, timeout, on_behalf_of) = script_path_to_payload(
script_path.to_path(),
Some(db_authed),
db.clone(),
&w_id,
run_query.skip_preprocessor,
)
.await?;
let tag = run_query.tag.clone().or(tag);
check_tag_available_for_workspace(&db, &w_id, &tag, &authed).await?;
@@ -4764,7 +4550,9 @@ pub async fn run_wait_result_script_by_path_internal(
tx.commit().await?;
let wait_result = run_wait_result(&db, uuid, &w_id, None, &authed.username).await;
handle_delete_after_completion(&db, uuid, &w_id, delete_after_use, delete_after_secs).await?;
if delete_after_use.unwrap_or(false) {
delete_job_metadata_after_use(&db, uuid).await?;
}
return wait_result;
}
@@ -4802,7 +4590,6 @@ pub async fn run_wait_result_script_by_hash(
dedicated_worker,
priority,
delete_after_use,
delete_after_secs,
timeout,
has_preprocessor,
on_behalf_of_email,
@@ -4889,7 +4676,9 @@ pub async fn run_wait_result_script_by_hash(
tx.commit().await?;
let wait_result = run_wait_result(&db, uuid, &w_id, None, &authed.username).await;
handle_delete_after_completion(&db, uuid, &w_id, delete_after_use, delete_after_secs).await?;
if delete_after_use.unwrap_or(false) {
delete_job_metadata_after_use(&db, uuid).await?;
}
return wait_result;
}
@@ -5059,7 +4848,7 @@ pub async fn stream_job(
(uuid, None)
}
RunnableId::ScriptId(ScriptId::ScriptHash(script_hash)) => {
let (uuid, _, _) = run_job_by_hash_inner(
let (uuid, _) = run_job_by_hash_inner(
authed.clone(),
db.clone(),
user_db,
@@ -6542,7 +6331,7 @@ pub async fn run_job_by_hash(
)
.await?;
let (uuid, _, _) = run_job_by_hash_inner(
let (uuid, _) = run_job_by_hash_inner(
authed,
db,
user_db,
@@ -6566,7 +6355,7 @@ pub async fn run_job_by_hash_inner(
run_query: RunJobQuery,
args: PushArgsOwned,
trigger: Option<TriggerMetadata>,
) -> error::Result<(Uuid, Option<bool>, Option<i32>)> {
) -> error::Result<(Uuid, Option<bool>)> {
#[cfg(feature = "enterprise")]
check_license_key_valid().await?;
@@ -6587,7 +6376,6 @@ pub async fn run_job_by_hash_inner(
on_behalf_of_email,
created_by,
delete_after_use,
delete_after_secs,
labels,
..
} = get_script_info_for_hash(Some(userdb_authed), &db, &w_id, hash)
@@ -6668,7 +6456,7 @@ pub async fn run_job_by_hash_inner(
.await?;
tx.commit().await?;
Ok((uuid, delete_after_use, delete_after_secs))
Ok((uuid, delete_after_use))
}
async fn get_log_file(Path((_w_id, file_p)): Path<(String, String)>) -> error::Result<Response> {

View File

@@ -131,7 +131,6 @@ pub mod oauth2_oss;
#[cfg(feature = "private")]
pub mod oidc_ee;
mod oidc_oss;
mod path_autocomplete;
mod raw_apps;
mod resources;
#[cfg(feature = "private")]
@@ -602,10 +601,6 @@ pub async fn run_server(
})
.nest("/ai", ai::workspaced_service())
.nest("/npm_proxy", windmill_api_npm_proxy::workspaced_service())
.nest(
"/path_autocomplete",
path_autocomplete::workspaced_service(),
)
.nest("/raw_apps", raw_apps::workspaced_service())
.nest("/resources", resources::workspaced_service())
.nest("/schedules", windmill_api_schedule::workspaced_service())

View File

@@ -1,86 +0,0 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2026
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
use std::{
sync::{Arc, LazyLock},
time::{Duration, Instant},
};
use axum::{
extract::{Extension, Path},
routing::get,
Json, Router,
};
use serde::Serialize;
use windmill_common::error::JsonResult;
use crate::db::{ApiAuthed, DB};
// Per-table row cap is inlined into the SQL as `LIMIT 5000`.
// With 6 tables, the absolute ceiling is ~30k paths pre-dedup.
/// Final cap applied after dedup/sort.
const MAX_PATHS: usize = 20_000;
/// TTL for the per-workspace path list cache.
const CACHE_TTL: Duration = Duration::from_secs(60);
/// Workspace-wide path list cache keyed by workspace_id only.
/// One entry per workspace shared across all users — autocomplete is a
/// navigation hint, not an access gate. Saves memory and warms faster.
static PATHS_CACHE: LazyLock<quick_cache::sync::Cache<String, (Arc<Vec<String>>, Instant)>> =
LazyLock::new(|| quick_cache::sync::Cache::new(500));
pub fn workspaced_service() -> Router {
Router::new().route("/list_paths", get(list_paths))
}
#[derive(Serialize)]
struct ListPathsResponse {
paths: Arc<Vec<String>>,
}
async fn list_paths(
_authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<ListPathsResponse> {
if let Some((cached, cached_at)) = PATHS_CACHE.get(&w_id) {
if cached_at.elapsed() < CACHE_TTL {
return Ok(Json(ListPathsResponse { paths: cached }));
}
PATHS_CACHE.remove(&w_id);
}
let mut paths: Vec<String> = sqlx::query_scalar!(
r#"
SELECT path AS "path!" FROM (
(SELECT DISTINCT path FROM script WHERE workspace_id = $1 AND archived = false AND deleted = false AND draft_only IS NOT true LIMIT 5000)
UNION
(SELECT path FROM flow WHERE workspace_id = $1 AND archived = false AND draft_only IS NOT true LIMIT 5000)
UNION
(SELECT path FROM app WHERE workspace_id = $1 LIMIT 5000)
UNION
(SELECT path FROM raw_app WHERE workspace_id = $1 LIMIT 5000)
UNION
(SELECT path FROM variable WHERE workspace_id = $1 LIMIT 5000)
UNION
(SELECT path FROM resource WHERE workspace_id = $1 LIMIT 5000)
) t
"#,
&w_id,
)
.fetch_all(&db)
.await?;
paths.sort_unstable();
paths.truncate(MAX_PATHS);
let paths = Arc::new(paths);
PATHS_CACHE.insert(w_id, (paths.clone(), Instant::now()));
Ok(Json(ListPathsResponse { paths }))
}

View File

@@ -28,10 +28,7 @@ use windmill_common::secret_backend::{database::DatabaseBackend, SecretBackend};
#[cfg(all(feature = "private", feature = "enterprise"))]
use windmill_common::{
global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING},
secret_backend::{
AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend,
AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings,
},
secret_backend::{SecretBackendConfig, VaultBackend, VaultSettings},
};
#[cfg(all(feature = "private", feature = "enterprise"))]
@@ -50,30 +47,6 @@ lazy_static::lazy_static! {
static ref VAULT_BACKEND_CACHE: RwLock<Option<CachedVaultBackend>> = RwLock::new(None);
}
// Cached Azure Key Vault backend
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAzureKvBackend {
backend: Arc<dyn SecretBackend>,
settings: AzureKeyVaultSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AZURE_KV_BACKEND_CACHE: RwLock<Option<CachedAzureKvBackend>> = RwLock::new(None);
}
// Cached AWS Secrets Manager backend
#[cfg(all(feature = "private", feature = "enterprise"))]
struct CachedAwsSmBackend {
backend: Arc<dyn SecretBackend>,
settings: AwsSecretsManagerSettings,
}
#[cfg(all(feature = "private", feature = "enterprise"))]
lazy_static::lazy_static! {
static ref AWS_SM_BACKEND_CACHE: RwLock<Option<CachedAwsSmBackend>> = RwLock::new(None);
}
/// Get the current secret backend based on global settings (EE only)
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
@@ -87,12 +60,6 @@ async fn get_secret_backend(db: &DB) -> Result<Arc<dyn SecretBackend>> {
SecretBackendConfig::HashiCorpVault(settings) => {
get_or_create_vault_backend(db, settings).await
}
SecretBackendConfig::AzureKeyVault(settings) => {
get_or_create_azure_kv_backend(db, settings).await
}
SecretBackendConfig::AwsSecretsManager(settings) => {
get_or_create_aws_sm_backend(db, settings).await
}
}
}
@@ -141,73 +108,7 @@ async fn get_or_create_vault_backend(
Ok(backend)
}
/// Get a cached Azure Key Vault backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_azure_kv_backend(
_db: &DB,
settings: AzureKeyVaultSettings,
) -> Result<Arc<dyn SecretBackend>> {
// Check if we have a cached backend with matching settings (read lock)
{
let cache = AZURE_KV_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
// Need to create a new backend - acquire write lock
let mut cache = AZURE_KV_BACKEND_CACHE.write().await;
// Double-check (another task may have created it while we waited)
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
// Create new backend
let backend: Arc<dyn SecretBackend> = Arc::new(AzureKeyVaultBackend::new(settings.clone()));
// Cache it
*cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Get a cached AWS SM backend or create a new one if settings changed
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn get_or_create_aws_sm_backend(
_db: &DB,
settings: AwsSecretsManagerSettings,
) -> Result<Arc<dyn SecretBackend>> {
{
let cache = AWS_SM_BACKEND_CACHE.read().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
}
let mut cache = AWS_SM_BACKEND_CACHE.write().await;
if let Some(ref cached) = *cache {
if cached.settings == settings {
return Ok(cached.backend.clone());
}
}
let backend: Arc<dyn SecretBackend> =
Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?);
*cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings });
Ok(backend)
}
/// Check if an external secret backend is currently configured (EE only)
/// Check if a Vault backend is currently configured (EE only)
#[cfg(all(feature = "private", feature = "enterprise"))]
async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? {
@@ -215,10 +116,7 @@ async fn is_vault_backend_configured(db: &DB) -> Result<bool> {
None => SecretBackendConfig::default(),
};
Ok(matches!(
config,
SecretBackendConfig::HashiCorpVault(_) | SecretBackendConfig::AzureKeyVault(_) | SecretBackendConfig::AwsSecretsManager(_)
))
Ok(matches!(config, SecretBackendConfig::HashiCorpVault(_)))
}
/// Check if a value is stored in Vault (indicated by the $vault: prefix)
@@ -227,24 +125,6 @@ fn is_vault_stored_value(value: &str) -> bool {
value.starts_with("$vault:")
}
/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix)
#[cfg(all(feature = "private", feature = "enterprise"))]
fn is_azure_kv_stored_value(value: &str) -> bool {
value.starts_with("$azure_kv:")
}
/// Check if a value is stored in AWS Secrets Manager
#[cfg(all(feature = "private", feature = "enterprise"))]
fn is_aws_sm_stored_value(value: &str) -> bool {
value.starts_with("$aws_sm:")
}
/// Check if a value is stored in any external secret backend
#[cfg(all(feature = "private", feature = "enterprise"))]
fn is_external_stored_value(value: &str) -> bool {
is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value)
}
/// Bulk rename secrets in Vault when a path prefix changes (e.g., user rename)
/// EE only feature.
///
@@ -270,7 +150,7 @@ pub async fn rename_vault_secrets_with_prefix(
new_prefix: &str,
variables: Vec<(String, String)>, // (path, value) pairs
) -> Result<Vec<(String, String)>> {
// Only process if an external secret backend is configured
// Only process if Vault is configured
if !is_vault_backend_configured(db).await? {
return Ok(vec![]);
}
@@ -279,20 +159,11 @@ pub async fn rename_vault_secrets_with_prefix(
let mut updates = Vec::new();
for (old_path, value) in variables {
// Only handle externally-stored values
if !is_external_stored_value(&value) {
// Only handle Vault-stored values
if !is_vault_stored_value(&value) {
continue;
}
// Determine the marker prefix from the stored value
let marker_prefix = if is_azure_kv_stored_value(&value) {
"$azure_kv:"
} else if is_aws_sm_stored_value(&value) {
"$aws_sm:"
} else {
"$vault:"
};
// Calculate new path by replacing prefix
let new_path = if old_path.starts_with(old_prefix) {
format!("{}{}", new_prefix, &old_path[old_prefix.len()..])
@@ -305,7 +176,7 @@ pub async fn rename_vault_secrets_with_prefix(
Ok(v) => v,
Err(Error::NotFound(_)) => {
// Just update DB reference
updates.push((old_path, format!("{}{}", marker_prefix, new_path)));
updates.push((old_path, format!("$vault:{}", new_path)));
continue;
}
Err(e) => {
@@ -340,7 +211,7 @@ pub async fn rename_vault_secrets_with_prefix(
);
}
updates.push((old_path, format!("{}{}", marker_prefix, new_path)));
updates.push((old_path, format!("$vault:{}", new_path)));
}
Ok(updates)

View File

@@ -294,13 +294,13 @@ async fn update_username_in_workpsace<'c>(
let old_prefix = format!("u/{}/", old_username);
let new_prefix = format!("u/{}/", new_username);
// Fetch all externally-stored secret variables under this user's path
// Fetch all Vault-stored secret variables under this user's path
let vault_secrets: Vec<(String, String)> = sqlx::query!(
r#"SELECT path, value FROM variable
WHERE path LIKE ('u/' || $1 || '/%')
AND workspace_id = $2
AND is_secret = true
AND (value LIKE '$vault:%' OR value LIKE '$azure_kv:%')"#,
AND value LIKE '$vault:%'"#,
old_username,
w_id
)

View File

@@ -84,7 +84,7 @@ struct ScriptMetadata {
#[serde(skip_serializing_if = "Option::is_none")]
pub timeout: Option<i32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub delete_after_secs: Option<i32>,
pub delete_after_use: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub restart_unless_cancelled: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -508,7 +508,7 @@ pub(crate) async fn tarball_workspace(
priority: script.priority,
tag: script.tag,
timeout: script.timeout,
delete_after_secs: script.delete_after_secs,
delete_after_use: script.delete_after_use,
restart_unless_cancelled: script.restart_unless_cancelled,
visible_to_runner_only: script.visible_to_runner_only,
auto_kind: script.auto_kind,

View File

@@ -9,7 +9,7 @@ default = []
enterprise = ["dep:aws-config"]
instance_config_schema = ["dep:schemars"]
local_reports = ["dep:rsa", "dep:aes-gcm"]
private = ["dep:aws-sdk-rds", "dep:aws-sdk-secretsmanager"]
private = ["dep:aws-sdk-rds"]
jemalloc = ["dep:tikv-jemalloc-ctl"]
tantivy = []
prometheus = ["dep:prometheus"]
@@ -80,7 +80,6 @@ postgres-native-tls.workspace = true
native-tls.workspace = true
aws-smithy-types-convert = { workspace = true, optional = true }
aws-sdk-secretsmanager = { workspace = true, optional = true }
aws-sdk-rds = { workspace = true, optional = true }
indexmap.workspace = true
bytes.workspace = true

View File

@@ -20,10 +20,6 @@ where
lazy_static::lazy_static! {
static ref OPENAI_AZURE_BASE_PATH: Option<String> = std::env::var("OPENAI_AZURE_BASE_PATH").ok();
static ref ALLOW_PRIVATE_AI_BASE_URLS: bool = std::env::var("ALLOW_PRIVATE_AI_BASE_URLS")
.ok()
.map(|v| v == "true" || v == "1")
.unwrap_or(false);
}
pub const OPENAI_BASE_URL: &str = "https://api.openai.com/v1";
@@ -64,16 +60,6 @@ impl AIProvider {
/// Get the base URL for the AI provider
pub async fn get_base_url(&self, resource_base_url: Option<String>, db: &DB) -> Result<String> {
if let Some(base_url) = resource_base_url {
if !*ALLOW_PRIVATE_AI_BASE_URLS {
crate::ssrf::validate_url_for_ssrf(&base_url)
.await
.map_err(|e| {
Error::BadRequest(format!(
"{e}. If you need to use private/internal AI endpoints, \
set the ALLOW_PRIVATE_AI_BASE_URLS=true environment variable"
))
})?;
}
return Ok(base_url);
}

View File

@@ -899,7 +899,7 @@ const SENSITIVE_SETTINGS: &[&str] = &[
/// Maps a top-level key to the sub-field names that must be redacted.
const NESTED_SENSITIVE_FIELDS: &[(&str, &[&str])] = &[
("smtp_settings", &["smtp_password"]),
("secret_backend", &["token", "client_secret", "secret_access_key"]),
("secret_backend", &["token"]),
(
"object_store_cache_config",
&["secret_key", "serviceAccountKey"],

View File

@@ -43,39 +43,6 @@ pub fn get_has_preprocessor_from_content_and_lang(
Ok(has_preprocessor)
}
pub async fn schedule_job_deletion(
db: &DB,
job_id: uuid::Uuid,
w_id: &str,
delete_after_secs: i32,
) -> crate::error::Result<()> {
sqlx::query!(
"INSERT INTO job_delete_schedule (job_id, workspace_id, delete_at) \
VALUES ($1, $2, now() + make_interval(secs => $3::double precision)) \
ON CONFLICT (job_id) DO NOTHING",
job_id,
w_id,
delete_after_secs as f64,
)
.execute(db)
.await?;
Ok(())
}
/// Resolve effective delete behavior from delete_after_use (bool) and delete_after_secs.
/// Returns Some(secs) if deletion should happen, None otherwise.
pub fn resolve_delete_after_secs(
delete_after_use: Option<bool>,
delete_after_secs: Option<i32>,
) -> Option<i32> {
match (delete_after_use, delete_after_secs) {
(_, Some(secs)) if secs >= 0 => Some(secs),
(_, Some(_)) => None, // reject negative values
(Some(true), None) => Some(0), // backward compat: immediate
_ => None,
}
}
pub async fn script_path_to_payload<'e>(
script_path: &str,
db_authed: Option<UserDbWithAuthed<'e, AuthedRef<'e>>>,
@@ -87,94 +54,88 @@ pub async fn script_path_to_payload<'e>(
Option<Tag>,
Option<bool>,
Option<i32>,
Option<i32>,
Option<OnBehalfOf>,
)> {
let (job_payload, tag, delete_after_use, delete_after_secs, script_timeout, on_behalf_of) =
if script_path.starts_with("hub/") {
let hub_script =
get_full_hub_script_by_path(StripPath(script_path.to_string()), &HTTP_CLIENT, None)
.await?;
let (job_payload, tag, delete_after_use, script_timeout, on_behalf_of) = if script_path
.starts_with("hub/")
{
let hub_script =
get_full_hub_script_by_path(StripPath(script_path.to_string()), &HTTP_CLIENT, None)
.await?;
let has_preprocessor = get_has_preprocessor_from_content_and_lang(
&hub_script.content,
&hub_script.language,
)?;
let has_preprocessor =
get_has_preprocessor_from_content_and_lang(&hub_script.content, &hub_script.language)?;
(
JobPayload::ScriptHub {
path: script_path.to_owned(),
apply_preprocessor: has_preprocessor && !skip_preprocessor.unwrap_or(false),
(
JobPayload::ScriptHub {
path: script_path.to_owned(),
apply_preprocessor: has_preprocessor && !skip_preprocessor.unwrap_or(false),
},
None,
None,
None,
None,
)
} else {
let ScriptHashInfo {
hash,
tag,
runnable_settings:
super::scripts::ScriptRunnableSettingsInline {
concurrency_settings,
debouncing_settings,
},
None,
None,
None,
None,
None,
)
cache_ttl,
cache_ignore_s3_path,
language,
dedicated_worker,
priority,
delete_after_use,
timeout,
has_preprocessor,
on_behalf_of_email,
created_by,
labels,
..
} = get_latest_deployed_hash_for_path(db_authed, db.clone(), w_id, script_path)
.await?
.prefetch_cached(&db)
.await?;
let on_behalf_of = if let Some(email) = on_behalf_of_email {
Some(OnBehalfOf {
email,
permissioned_as: username_to_permissioned_as(created_by.as_str()),
})
} else {
let ScriptHashInfo {
hash,
tag,
runnable_settings:
super::scripts::ScriptRunnableSettingsInline {
concurrency_settings,
debouncing_settings,
},
None
};
(
JobPayload::ScriptHash {
hash: ScriptHash(hash),
path: script_path.to_owned(),
cache_ttl,
cache_ignore_s3_path,
language,
dedicated_worker,
priority,
delete_after_use,
delete_after_secs,
timeout,
has_preprocessor,
on_behalf_of_email,
created_by,
apply_preprocessor: !skip_preprocessor.unwrap_or(false)
&& has_preprocessor.unwrap_or(false),
debouncing_settings,
concurrency_settings,
labels,
..
} = get_latest_deployed_hash_for_path(db_authed, db.clone(), w_id, script_path)
.await?
.prefetch_cached(&db)
.await?;
let on_behalf_of = if let Some(email) = on_behalf_of_email {
Some(OnBehalfOf {
email,
permissioned_as: username_to_permissioned_as(created_by.as_str()),
})
} else {
None
};
(
JobPayload::ScriptHash {
hash: ScriptHash(hash),
path: script_path.to_owned(),
cache_ttl,
cache_ignore_s3_path,
language,
dedicated_worker,
priority,
apply_preprocessor: !skip_preprocessor.unwrap_or(false)
&& has_preprocessor.unwrap_or(false),
debouncing_settings,
concurrency_settings,
labels,
},
tag,
delete_after_use,
delete_after_secs,
timeout,
on_behalf_of,
)
};
},
tag,
delete_after_use,
timeout,
on_behalf_of,
)
};
Ok((
job_payload,
tag,
delete_after_use,
delete_after_secs,
script_timeout,
on_behalf_of,
))
@@ -185,7 +146,7 @@ pub async fn get_payload_tag_from_prefixed_path(
db: &DB,
w_id: &str,
) -> Result<(JobPayload, Option<String>, Option<OnBehalfOf>), Error> {
let (payload, tag, _, _, _, on_behalf_of) = if path.starts_with("script/") {
let (payload, tag, _, _, on_behalf_of) = if path.starts_with("script/") {
script_path_to_payload(
path.strip_prefix("script/").unwrap(),
None,
@@ -209,7 +170,6 @@ pub async fn get_payload_tag_from_prefixed_path(
None,
None,
None,
None,
)
} else {
let FlowVersionInfo { dedicated_worker, tag, version, labels, .. } =
@@ -226,7 +186,6 @@ pub async fn get_payload_tag_from_prefixed_path(
None,
None,
None,
None,
)
}
} else {

View File

@@ -401,7 +401,7 @@ pub struct PrepareQueryResult {
pub error: Option<String>,
}
#[derive(Deserialize, Serialize, Clone)]
#[derive(Deserialize, Serialize)]
pub struct PgDatabase {
pub host: String,
pub user: Option<String>,
@@ -632,180 +632,6 @@ impl PgDatabase {
}
}
/// Validate a database name to prevent SQL injection.
/// Must start with a letter, contain only alphanumeric characters or underscores, and be <= 63 chars.
pub fn validate_dbname(dbname: &str) -> error::Result<()> {
let dbname = dbname.trim();
if dbname.is_empty() {
return Err(error::Error::BadRequest(
"Database name cannot be empty".to_string(),
));
}
if dbname.len() > 63 {
return Err(error::Error::BadRequest(
"Database name cannot exceed 63 characters".to_string(),
));
}
if !dbname
.chars()
.next()
.map_or(false, |c| c.is_ascii_alphabetic())
{
return Err(error::Error::BadRequest(
"Database name must start with a letter".to_string(),
));
}
if !dbname
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_')
{
return Err(error::Error::BadRequest(
"Database name must contain only alphanumeric characters or underscores".to_string(),
));
}
Ok(())
}
/// Drop a custom instance database: validate, terminate connections, DROP DATABASE, remove from global_settings.
pub async fn drop_custom_instance_database(db: &DB, dbname: &str) -> error::Result<()> {
let dbname = dbname.trim();
validate_dbname(dbname)?;
let wmill_pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
if wmill_pg_creds.dbname.trim().eq_ignore_ascii_case(dbname) {
return Err(error::Error::BadRequest(
"Cannot drop the main Windmill database".to_string(),
));
}
let db_exists = sqlx::query_scalar!(
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_database WHERE datname = $1)",
dbname
)
.fetch_one(db)
.await?
.unwrap_or(false);
if db_exists {
// Terminate active connections
if let Err(e) = sqlx::query(&format!(
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '{}' AND pid <> pg_backend_pid()",
dbname.replace('\'', "''")
))
.execute(db)
.await
{
tracing::warn!("Failed to terminate connections to '{}': {}", dbname, e);
}
// Drop the database
sqlx::query(&format!("DROP DATABASE IF EXISTS \"{}\"", dbname))
.execute(db)
.await
.map_err(|e| {
error::Error::internal_err(format!("Failed to drop database '{}': {}", dbname, e))
})?;
tracing::info!("Dropped instance database '{}'", dbname);
} else {
tracing::info!("Database '{}' does not exist, skipping drop", dbname);
}
// Always remove from global_settings
sqlx::query!(
r#"UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'"#,
dbname
)
.execute(db)
.await?;
Ok(())
}
/// Create a custom instance database: CREATE DATABASE, grant permissions, register in global_settings.
/// The `tag` is stored in global_settings metadata (e.g. "datatable" or "ducklake").
pub async fn create_custom_instance_database(
db: &DB,
dbname: &str,
tag: &str,
) -> error::Result<()> {
let dbname = dbname.trim();
validate_dbname(dbname)?;
let db_exists = sqlx::query_scalar!(
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_database WHERE datname = $1)",
dbname
)
.fetch_one(db)
.await?
.unwrap_or(false);
if db_exists {
return Err(error::Error::BadRequest(format!(
"Database '{}' already exists",
dbname
)));
}
sqlx::query(&format!("CREATE DATABASE \"{}\"", dbname))
.execute(db)
.await
.map_err(|e| {
error::Error::internal_err(format!("Failed to create database '{}': {}", dbname, e))
})?;
// Grant permissions to custom_instance_user
let wmill_pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
let new_pg_creds = PgDatabase { dbname: dbname.to_string(), ..wmill_pg_creds };
let (client, connection) = new_pg_creds.connect().await?;
let join_handle = tokio::spawn(async move { connection.await });
if let Err(e) = client
.batch_execute(&format!(
"GRANT CONNECT ON DATABASE \"{dbname}\" TO custom_instance_user;
GRANT USAGE ON SCHEMA public TO custom_instance_user;
GRANT CREATE ON SCHEMA public TO custom_instance_user;
GRANT CREATE ON DATABASE \"{dbname}\" TO custom_instance_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO custom_instance_user;"
))
.await
{
tracing::warn!(
"Failed to grant permissions on '{}': {}. Continuing.",
dbname,
e
);
}
drop(client);
join_handle
.await
.map_err(|e| error::Error::internal_err(format!("join error: {}", e)))?
.map_err(|e| error::Error::internal_err(format!("tokio_postgres error: {}", e)))?;
// Register in global_settings
let status_json = serde_json::json!({
"logs": {
"created_database": "OK",
"db_connect": "OK",
"grant_permissions": "OK"
},
"success": true,
"error": null,
"tag": tag
});
sqlx::query!(
r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (COALESCE(value->'databases', '{}'::jsonb) || to_jsonb($1::json))) WHERE name = 'custom_instance_pg_databases'"#,
serde_json::json!({ (dbname): status_json })
)
.execute(db)
.await?;
tracing::info!("Created custom instance database '{}'", dbname);
Ok(())
}
#[derive(Clone)]
pub enum DatabaseUrl {
#[cfg(all(feature = "enterprise", feature = "private"))]
@@ -1037,7 +863,6 @@ pub struct ScriptHashInfo<SR> {
pub dedicated_worker: Option<bool>,
pub priority: Option<i16>,
pub delete_after_use: Option<bool>,
pub delete_after_secs: Option<i32>,
pub timeout: Option<i32>,
pub has_preprocessor: Option<bool>,
pub on_behalf_of_email: Option<String>,
@@ -1068,7 +893,6 @@ impl ScriptHashInfo<ScriptRunnableSettingsHandle> {
dedicated_worker: self.dedicated_worker,
priority: self.priority,
delete_after_use: self.delete_after_use,
delete_after_secs: self.delete_after_secs,
timeout: self.timeout,
has_preprocessor: self.has_preprocessor,
on_behalf_of_email: self.on_behalf_of_email,
@@ -1238,7 +1062,6 @@ async fn get_script_info_for_hash_inner<'e, E: sqlx::PgExecutor<'e>>(
dedicated_worker,
priority,
delete_after_use,
delete_after_secs,
timeout,
has_preprocessor,
on_behalf_of_email,

View File

@@ -1264,8 +1264,6 @@ fn get_user_default_value(column: &ColumnDef) -> Option<String> {
if let Some(ref val) = column.default_user_value {
if is_truthy(val) {
return match val {
// SAFETY: not escaped — this SQL runs against the user's own external database
// (DB Studio), not Windmill's internal DB. Users can already run arbitrary SQL.
serde_json::Value::String(s) => Some(format!("'{}'", s)),
other => Some(other.to_string()),
};
@@ -1751,8 +1749,6 @@ fn format_default_value(s: &str, datatype: &str, db_type: DbType) -> String {
return s[1..s.len() - 1].to_string();
}
let escaped = escape_sql_literal(s);
// SAFETY: datatype is not escaped — this SQL runs against the user's own external database
// (DB Studio), not Windmill's internal DB. Users can already run arbitrary SQL.
if db_type == DbType::Postgresql {
return format!("CAST('{}' AS {})", escaped, datatype);
}
@@ -1831,8 +1827,6 @@ fn render_fk_ddl(
quote_table_name(&target_table_raw, db_type),
target_quoted.join(", ")
));
// SAFETY: on_delete/on_update are not validated — this SQL runs against the user's own
// external database (DB Studio), not Windmill's internal DB. Users can already run arbitrary SQL.
if fk.on_delete != "NO ACTION" {
sql.push_str(&format!(" ON DELETE {}", fk.on_delete));
}
@@ -1969,8 +1963,6 @@ fn render_fk_inline(fk: &TableEditorForeignKey, use_schema: bool, db_type: DbTyp
target_table,
target_cols.join(", ")
);
// SAFETY: on_delete/on_update are not validated — this SQL runs against the user's own
// external database (DB Studio), not Windmill's internal DB. Users can already run arbitrary SQL.
if fk.on_delete != "NO ACTION" {
sql.push_str(&format!(" ON DELETE {}", fk.on_delete));
}
@@ -2214,8 +2206,6 @@ fn render_alter_column(
Ok(queries)
}
// SAFETY: datatype is not escaped — this SQL runs against the user's own external database
// (DB Studio), not Windmill's internal DB. Users can already run arbitrary SQL.
fn render_alter_datatype(table_ref: &str, col: &str, datatype: &str, db_type: DbType) -> String {
let qc = qi(col, db_type);
match db_type {
@@ -4471,224 +4461,3 @@ mod tests {
assert_eq!(sql, "SHOW PRIMARY KEYS IN ACCOUNT");
}
}
// ============================================================================
// Full schema introspection types and logic (used by get_datatable_full_schema)
// ============================================================================
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct FullSchemaColumn {
pub name: String,
pub datatype: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub primary_key: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub default_value: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub nullable: Option<bool>,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct FullSchemaForeignKeyColumn {
#[serde(skip_serializing_if = "Option::is_none")]
pub source_column: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub target_column: Option<String>,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct FullSchemaForeignKey {
#[serde(skip_serializing_if = "Option::is_none")]
pub target_table: Option<String>,
pub columns: Vec<FullSchemaForeignKeyColumn>,
pub on_delete: String,
pub on_update: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub fk_constraint_name: Option<String>,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct FullSchemaTable {
pub name: String,
pub columns: Vec<FullSchemaColumn>,
pub foreign_keys: Vec<FullSchemaForeignKey>,
#[serde(skip_serializing_if = "Option::is_none")]
pub pk_constraint_name: Option<String>,
}
/// Full database schema: { schema_name: { table_name: FullSchemaTable } }
pub type FullDatabaseSchema =
std::collections::HashMap<String, std::collections::HashMap<String, FullSchemaTable>>;
fn pg_action_to_string(action: &str) -> String {
match action {
"a" => "NO ACTION".to_string(),
"r" => "RESTRICT".to_string(),
"c" => "CASCADE".to_string(),
"n" => "SET NULL".to_string(),
"d" => "SET DEFAULT".to_string(),
_ => "NO ACTION".to_string(),
}
}
/// Introspect a PostgreSQL database and return the full schema.
/// Takes a connected tokio_postgres Client.
pub async fn pg_get_full_schema(
client: &tokio_postgres::Client,
) -> Result<FullDatabaseSchema, String> {
let column_rows = client
.query(
"SELECT
ns.nspname AS schema_name,
c.relname AS table_name,
a.attname AS column_name,
pg_catalog.format_type(a.atttypid, a.atttypmod) AS datatype,
(SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid, true) for 128)
FROM pg_catalog.pg_attrdef d
WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef) AS default_value,
CASE a.attnotnull WHEN false THEN true ELSE false END AS nullable,
EXISTS (
SELECT 1 FROM pg_catalog.pg_index i
WHERE i.indrelid = c.oid AND i.indisprimary AND a.attnum = ANY(i.indkey)
) AS is_primary_key,
(SELECT con.conname FROM pg_catalog.pg_constraint con
WHERE con.conrelid = c.oid AND con.contype = 'p' LIMIT 1) AS pk_constraint_name
FROM pg_catalog.pg_attribute a
JOIN pg_catalog.pg_class c ON a.attrelid = c.oid
JOIN pg_catalog.pg_namespace ns ON c.relnamespace = ns.oid
WHERE c.relkind = 'r'
AND a.attnum > 0
AND NOT a.attisdropped
AND ns.nspname NOT IN ('pg_catalog', 'information_schema')
ORDER BY ns.nspname, c.relname, a.attnum",
&[],
)
.await
.map_err(|e| format!("Failed to query columns: {}", e))?;
let fk_rows = client
.query(
"SELECT
ns.nspname AS schema_name,
c.relname AS table_name,
con.conname AS fk_constraint_name,
att_src.attname AS source_column,
ns_ref.nspname AS ref_schema,
c_ref.relname AS ref_table,
att_ref.attname AS ref_column,
con.confdeltype::text AS on_delete,
con.confupdtype::text AS on_update
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class c ON con.conrelid = c.oid
JOIN pg_catalog.pg_namespace ns ON c.relnamespace = ns.oid
JOIN pg_catalog.pg_class c_ref ON con.confrelid = c_ref.oid
JOIN pg_catalog.pg_namespace ns_ref ON c_ref.relnamespace = ns_ref.oid
CROSS JOIN LATERAL unnest(con.conkey, con.confkey) WITH ORDINALITY AS u(src_attnum, ref_attnum, ord)
JOIN pg_catalog.pg_attribute att_src ON att_src.attrelid = c.oid AND att_src.attnum = u.src_attnum
JOIN pg_catalog.pg_attribute att_ref ON att_ref.attrelid = c_ref.oid AND att_ref.attnum = u.ref_attnum
WHERE con.contype = 'f'
AND ns.nspname NOT IN ('pg_catalog', 'information_schema')
ORDER BY ns.nspname, c.relname, con.conname, u.ord",
&[],
)
.await
.map_err(|e| format!("Failed to query foreign keys: {}", e))?;
let mut result: FullDatabaseSchema = std::collections::HashMap::new();
for row in &column_rows {
let schema_name: &str = row.get("schema_name");
let table_name: &str = row.get("table_name");
let column_name: &str = row.get("column_name");
let datatype: &str = row.get("datatype");
let default_value: Option<&str> = row.get("default_value");
let nullable: bool = row.get("nullable");
let is_primary_key: bool = row.get("is_primary_key");
let pk_constraint_name: Option<&str> = row.get("pk_constraint_name");
let schema_tables = result.entry(schema_name.to_string()).or_default();
let table = schema_tables
.entry(table_name.to_string())
.or_insert_with(|| FullSchemaTable {
name: table_name.to_string(),
columns: vec![],
foreign_keys: vec![],
pk_constraint_name: pk_constraint_name.map(|s| s.to_string()),
});
table.columns.push(FullSchemaColumn {
name: column_name.to_string(),
datatype: datatype.to_string(),
primary_key: if is_primary_key { Some(true) } else { None },
default_value: default_value.map(|s| s.to_string()),
nullable: Some(nullable),
});
}
let mut fk_map: std::collections::HashMap<
(String, String, String),
(
Option<String>,
Vec<FullSchemaForeignKeyColumn>,
String,
String,
),
> = std::collections::HashMap::new();
for row in &fk_rows {
let schema_name: &str = row.get("schema_name");
let table_name: &str = row.get("table_name");
let fk_name: &str = row.get("fk_constraint_name");
let source_column: &str = row.get("source_column");
let ref_schema: &str = row.get("ref_schema");
let ref_table: &str = row.get("ref_table");
let ref_column: &str = row.get("ref_column");
let on_delete: &str = row.get("on_delete");
let on_update: &str = row.get("on_update");
let target_table = if ref_schema == schema_name {
ref_table.to_string()
} else {
format!("{}.{}", ref_schema, ref_table)
};
let key = (
schema_name.to_string(),
table_name.to_string(),
fk_name.to_string(),
);
let entry = fk_map.entry(key).or_insert_with(|| {
(
Some(target_table.clone()),
vec![],
pg_action_to_string(on_delete),
pg_action_to_string(on_update),
)
});
entry.1.push(FullSchemaForeignKeyColumn {
source_column: Some(source_column.to_string()),
target_column: Some(ref_column.to_string()),
});
}
let mut fk_entries: Vec<_> = fk_map.into_iter().collect();
fk_entries.sort_by(|a, b| a.0.cmp(&b.0));
for ((schema_name, table_name, fk_name), (target_table, columns, on_delete, on_update)) in
fk_entries
{
if let Some(schema_tables) = result.get_mut(&schema_name) {
if let Some(table) = schema_tables.get_mut(&table_name) {
table.foreign_keys.push(FullSchemaForeignKey {
target_table,
columns,
on_delete,
on_update,
fk_constraint_name: Some(fk_name),
});
}
}
}
Ok(result)
}

View File

@@ -106,7 +106,6 @@ pub async fn prefetch_cached_script(
cache_ignore_s3_path: script.cache_ignore_s3_path,
timeout: script.timeout,
delete_after_use: script.delete_after_use,
delete_after_secs: script.delete_after_secs,
restart_unless_cancelled: script.restart_unless_cancelled,
visible_to_runner_only: script.visible_to_runner_only,
auto_kind: script.auto_kind,
@@ -347,7 +346,6 @@ pub async fn fetch_script_for_update<'a>(
cache_ignore_s3_path,
timeout,
delete_after_use,
delete_after_secs,
restart_unless_cancelled,
visible_to_runner_only,
auto_kind,
@@ -422,7 +420,6 @@ pub async fn clone_script<'c>(
priority: s.priority,
timeout: s.timeout,
delete_after_use: s.delete_after_use,
delete_after_secs: s.delete_after_secs,
restart_unless_cancelled: s.restart_unless_cancelled,
deployment_message,
visible_to_runner_only: s.visible_to_runner_only,
@@ -452,14 +449,14 @@ pub async fn clone_script<'c>(
created_by, schema, is_template, extra_perms, lock, language, kind, tag, \
draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, \
dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, \
delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, \
delete_after_use, timeout, concurrency_key, visible_to_runner_only, auto_kind, \
codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)
SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, \
content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, \
draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, \
dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, \
delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, \
delete_after_use, timeout, concurrency_key, visible_to_runner_only, auto_kind, \
codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels
FROM script WHERE hash = $2 AND workspace_id = $3;

View File

@@ -1,69 +0,0 @@
/*
* Author: Windmill Labs, Inc
* Copyright (C) Windmill Labs, Inc - All Rights Reserved
* Unauthorized copying of this file, via any medium is strictly prohibited.
*/
use async_trait::async_trait;
use crate::db::DB;
use crate::error::{Error, Result};
use super::{AwsSecretsManagerSettings, SecretBackend, SecretMigrationReport};
pub struct AwsSecretsManagerBackend;
impl AwsSecretsManagerBackend {
pub fn new(_settings: AwsSecretsManagerSettings) -> Self {
AwsSecretsManagerBackend
}
}
#[async_trait]
impl SecretBackend for AwsSecretsManagerBackend {
async fn get_secret(&self, _workspace_id: &str, _path: &str) -> Result<String> {
Err(Error::internal_err(
"AWS Secrets Manager integration requires Enterprise Edition".to_string(),
))
}
async fn set_secret(&self, _workspace_id: &str, _path: &str, _value: &str) -> Result<()> {
Err(Error::internal_err(
"AWS Secrets Manager integration requires Enterprise Edition".to_string(),
))
}
async fn delete_secret(&self, _workspace_id: &str, _path: &str) -> Result<()> {
Err(Error::internal_err(
"AWS Secrets Manager integration requires Enterprise Edition".to_string(),
))
}
fn backend_name(&self) -> &'static str {
"aws_secrets_manager"
}
}
pub async fn test_aws_sm_connection(_settings: &AwsSecretsManagerSettings) -> Result<()> {
Err(Error::internal_err(
"AWS Secrets Manager integration requires Enterprise Edition".to_string(),
))
}
pub async fn migrate_secrets_to_aws_sm(
_db: &DB,
_settings: &AwsSecretsManagerSettings,
) -> Result<SecretMigrationReport> {
Err(Error::internal_err(
"AWS Secrets Manager integration requires Enterprise Edition".to_string(),
))
}
pub async fn migrate_secrets_from_aws_sm(
_db: &DB,
_settings: &AwsSecretsManagerSettings,
) -> Result<SecretMigrationReport> {
Err(Error::internal_err(
"AWS Secrets Manager integration requires Enterprise Edition".to_string(),
))
}

View File

@@ -1,69 +0,0 @@
/*
* Author: Windmill Labs, Inc
* Copyright (C) Windmill Labs, Inc - All Rights Reserved
* Unauthorized copying of this file, via any medium is strictly prohibited.
*/
use async_trait::async_trait;
use crate::db::DB;
use crate::error::{Error, Result};
use super::{AzureKeyVaultSettings, SecretBackend, SecretMigrationReport};
pub struct AzureKeyVaultBackend;
impl AzureKeyVaultBackend {
pub fn new(_settings: AzureKeyVaultSettings) -> Self {
AzureKeyVaultBackend
}
}
#[async_trait]
impl SecretBackend for AzureKeyVaultBackend {
async fn get_secret(&self, _workspace_id: &str, _path: &str) -> Result<String> {
Err(Error::internal_err(
"Azure Key Vault integration requires Enterprise Edition".to_string(),
))
}
async fn set_secret(&self, _workspace_id: &str, _path: &str, _value: &str) -> Result<()> {
Err(Error::internal_err(
"Azure Key Vault integration requires Enterprise Edition".to_string(),
))
}
async fn delete_secret(&self, _workspace_id: &str, _path: &str) -> Result<()> {
Err(Error::internal_err(
"Azure Key Vault integration requires Enterprise Edition".to_string(),
))
}
fn backend_name(&self) -> &'static str {
"azure_key_vault"
}
}
pub async fn test_azure_kv_connection(_settings: &AzureKeyVaultSettings) -> Result<()> {
Err(Error::internal_err(
"Azure Key Vault integration requires Enterprise Edition".to_string(),
))
}
pub async fn migrate_secrets_to_azure_kv(
_db: &DB,
_settings: &AzureKeyVaultSettings,
) -> Result<SecretMigrationReport> {
Err(Error::internal_err(
"Azure Key Vault integration requires Enterprise Edition".to_string(),
))
}
pub async fn migrate_secrets_from_azure_kv(
_db: &DB,
_settings: &AzureKeyVaultSettings,
) -> Result<SecretMigrationReport> {
Err(Error::internal_err(
"Azure Key Vault integration requires Enterprise Edition".to_string(),
))
}

View File

@@ -18,13 +18,6 @@ pub mod database;
pub mod vault_ee;
pub mod vault_oss;
#[cfg(feature = "private")]
pub mod azure_kv_ee;
pub mod azure_kv_oss;
#[cfg(feature = "private")]
pub mod aws_sm_ee;
pub mod aws_sm_oss;
#[cfg(test)]
mod tests;
@@ -35,18 +28,6 @@ pub use vault_ee::*;
#[cfg(not(feature = "private"))]
pub use vault_oss::*;
#[cfg(feature = "private")]
pub use azure_kv_ee::*;
#[cfg(not(feature = "private"))]
pub use azure_kv_oss::*;
#[cfg(feature = "private")]
pub use aws_sm_ee::*;
#[cfg(not(feature = "private"))]
pub use aws_sm_oss::*;
use async_trait::async_trait;
use serde::{Deserialize, Serialize};
@@ -99,10 +80,6 @@ pub enum SecretBackendConfig {
Database,
/// Store secrets in HashiCorp Vault (Enterprise Edition only)
HashiCorpVault(VaultSettings),
/// Store secrets in Azure Key Vault (Enterprise Edition only)
AzureKeyVault(AzureKeyVaultSettings),
/// Store secrets in AWS Secrets Manager (Enterprise Edition only)
AwsSecretsManager(AwsSecretsManagerSettings),
}
impl Default for SecretBackendConfig {
@@ -131,42 +108,6 @@ pub struct VaultSettings {
pub token: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct AzureKeyVaultSettings {
/// Azure Key Vault URL (e.g., "https://myvault.vault.azure.net")
pub vault_url: String,
/// Azure AD tenant ID
pub tenant_id: String,
/// Azure AD application (client) ID
pub client_id: String,
/// Azure AD client secret
#[serde(skip_serializing_if = "Option::is_none")]
pub client_secret: Option<String>,
/// Static Bearer token for testing/development (optional)
/// If provided, this is used instead of OAuth2 client credentials authentication
#[serde(skip_serializing_if = "Option::is_none")]
pub token: Option<String>,
}
/// Settings for AWS Secrets Manager integration
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct AwsSecretsManagerSettings {
/// AWS region (e.g., "us-east-1")
pub region: String,
/// Static AWS access key ID (optional - uses default credential chain if not provided)
#[serde(skip_serializing_if = "Option::is_none")]
pub access_key_id: Option<String>,
/// Static AWS secret access key (optional)
#[serde(skip_serializing_if = "Option::is_none")]
pub secret_access_key: Option<String>,
/// Custom endpoint URL for LocalStack/testing (optional)
#[serde(skip_serializing_if = "Option::is_none")]
pub endpoint_url: Option<String>,
/// Prefix for secret names in AWS Secrets Manager (e.g., "windmill/")
#[serde(skip_serializing_if = "Option::is_none")]
pub prefix: Option<String>,
}
/// Result of a secret migration operation
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SecretMigrationReport {

Some files were not shown because too many files have changed in this diff Show More