test: add regression tests for approval_info form schema resolution

Tests verify:
- raw_flow path (FlowPreview): form_schema present ✓
- flow_node path (graph-based branch/loop): form_schema present ✓ (fails without fix)
- missing flow_node: form_schema absent ✓

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.agents/skills/native-trigger/SKILL.md

@@ -1,3 +1,8 @@
+---
+name: native-trigger
+description: Guidance for adding native trigger services to Windmill. Use when implementing or modifying native trigger integrations across the backend and frontend.
+---
+
 # Skill: Adding Native Trigger Services
 
 This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications.

.claude/review-prompt.md

@@ -0,0 +1,25 @@
+# Code Review Instructions
+
+Review this pull request and provide comprehensive feedback.
+
+## Focus Areas
+
+- **Code quality and best practices** — does the code follow established patterns?
+- **Potential bugs or issues** — will this code work correctly in all cases?
+- **Performance considerations** — are there unnecessary allocations, N+1 queries, or bottlenecks?
+- **Security implications** — injection, auth bypass, data exposure?
+
+## CLAUDE.md Compliance
+
+Read all relevant CLAUDE.md files (root and in directories containing changed files). Check each rule against the changed code. Quote the exact rule when flagging a violation.
+
+## Review Guidelines
+
+- Provide detailed feedback using inline comments for specific issues
+- Use top-level comments for general observations or praise
+- Only flag issues introduced by this PR, not pre-existing problems
+- Self-validate each finding: "Is this definitely a real issue?" If uncertain, discard it
+
+## Testing Instructions
+
+At the end of your review, add complete instructions to reproduce the added changes through the app interface. These instructions will be given to a tester so they can verify the changes. It should be a short descriptive text (not a step-by-step or a list) on how to navigate the app (what page, what action, what input, etc.) to see the changes.

.claude/skills/local-review/SKILL.md

@@ -6,53 +6,24 @@ description: Code review a pull request for bugs and CLAUDE.md compliance. MUST
 
 # Local Code Review Skill
 
-Review a pull request for real bugs and CLAUDE.md compliance violations. This review targets HIGH SIGNAL issues only.
-
-## Review Philosophy
-
-- **Only flag issues you are certain about.** If you are not sure an issue is real, do not flag it. False positives erode trust and waste reviewer time.
-- Think like a senior engineer doing a final review — flag things that would cause incidents, not things that are merely imperfect.
-
-## What to Flag
-
-- Code that won't compile or parse (syntax errors, type errors, missing imports)
-- Code that will definitely produce wrong results regardless of inputs
-- Clear, unambiguous CLAUDE.md violations (quote the exact rule being violated)
-- Security issues in introduced code (injection, auth bypass, data exposure)
-- Incorrect logic that will fail in production
-
-## What NOT to Flag
-
-- Code style or quality concerns
-- Potential issues that depend on specific inputs or runtime state
-- Subjective suggestions or improvements
-- Pre-existing issues not introduced by this PR
-- Pedantic nitpicks a senior engineer wouldn't flag
-- Issues a linter or type checker will catch
-- General quality concerns unless explicitly prohibited in CLAUDE.md
-- Issues silenced via lint ignore comments
+Run the same review locally that the GitHub Claude Auto Review action runs on PRs. The shared review instructions live in `.claude/review-prompt.md` — read that file first and follow its instructions.
 
 ## Execution Steps
 
-1. **Determine the PR scope**:
+1. **Read `.claude/review-prompt.md`** for the review criteria and focus areas
+
+2. **Determine the PR scope**:
    - If an argument is provided, use it as the PR number or branch
    - Otherwise, detect from the current branch vs main
    - Run `gh pr view` if a PR exists, or use `git diff main...HEAD`
 
-2. **Find relevant CLAUDE.md files**:
-   - Read the root `CLAUDE.md`
-   - Check for CLAUDE.md files in directories containing changed files
-
 3. **Get the diff and metadata**:
    - `gh pr diff` or `git diff main...HEAD` for the full diff
    - `gh pr view` or `git log main..HEAD --oneline` for context
 
 4. **Read changed files** where the diff alone is insufficient to understand context
 
-5. **Review for**:
-   - CLAUDE.md compliance — check each rule against the changed code
-   - Bugs and logic errors — will this code work correctly?
-   - Security issues — injection, auth, data exposure in new code
+5. **Apply the review instructions from `.claude/review-prompt.md`**
 
 6. **Self-validate each finding**: Before reporting, ask yourself:
    - "Is this definitely a real issue, not a false positive?"

.claude/skills/native-trigger/SKILL.md

@@ -1,3 +1,8 @@
+---
+name: native-trigger
+description: Guidance for adding native trigger services to Windmill. Use when implementing or modifying native trigger integrations across the backend and frontend.
+---
+
 # Skill: Adding Native Trigger Services
 
 This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications.

.claude/skills/pr/SKILL.md

@@ -61,12 +61,13 @@ Generated with [Claude Code](https://claude.com/claude-code)
 1. Run `git status` to check for uncommitted changes
 2. Run `git log main..HEAD --oneline` to see all commits in this branch
 3. Run `git diff main...HEAD` to see the full diff against main
-4. Check if remote branch exists and is up to date:
+4. **Run `/local-review`** before creating the PR. If issues are found, fix them and commit before proceeding. Do not skip this step.
+5. Check if remote branch exists and is up to date:
    ```bash
    git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
    ```
-5. Push to remote if needed: `git push -u origin HEAD`
-6. Create draft PR using gh CLI:
+6. Push to remote if needed: `git push -u origin HEAD`
+7. Create draft PR using gh CLI:
    ```bash
    gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
    ## Summary
@@ -85,7 +86,7 @@ Generated with [Claude Code](https://claude.com/claude-code)
    EOF
    )"
    ```
-7. Return the PR URL to the user
+8. Return the PR URL to the user
 
 ## EE Companion PR (when `*_ee.rs` files were modified)
 

.github/workflows/benchmark.yml

@@ -290,6 +290,49 @@ jobs:
           path: |
             *.json
 
+  benchmark_wac:
+    runs-on: ubicloud-standard-8
+    services:
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_DB: windmill
+          POSTGRES_PASSWORD: changeme
+          POSTGRES_INITDB_ARGS: "-c shared_buffers=2GB -c work_mem=32MB -c effective_cache_size=4GB"
+        options: >-
+          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
+          --health-retries 5
+          --shm-size=2g
+      windmill:
+        image: ghcr.io/windmill-labs/windmill-ee:main
+        env:
+          DATABASE_URL: postgres://postgres:changeme@postgres:5432/windmill
+          LICENSE_KEY: ${{ secrets.WM_LICENSE_KEY_CI }}
+          WORKER_GROUP: main
+          WORKER_TAGS: deno,bun,go,python3,bash,dependency,flow,nativets
+        options: >-
+          --pull always --health-interval 10s --health-timeout 5s
+          --health-retries 5 --health-cmd "curl
+          http://localhost:8000/api/version"
+        ports:
+          - 8000:8000
+    steps:
+      - uses: denoland/setup-deno@v2
+        with:
+          deno-version: v2.x
+      - name: benchmark
+        timeout-minutes: 30
+        run: deno run  -A -r
+          https://raw.githubusercontent.com/windmill-labs/windmill/${GITHUB_REF##ref/head/}/benchmarks/benchmark_suite.ts
+          -c
+          https://raw.githubusercontent.com/windmill-labs/windmill/${GITHUB_REF##ref/head/}/benchmarks/suite_wac.json
+      - name: Save benchmark results
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark_wac
+          path: |
+            *.json
+
   benchmark_graphs:
     runs-on: ubicloud
     needs:
@@ -297,6 +340,7 @@ jobs:
       - benchmark_dedicated
       - benchmark_4workers
       - benchmark_8workers
+      - benchmark_wac
     steps:
       - uses: denoland/setup-deno@v2
         with:

.github/workflows/cli-tests.yml

@@ -1,6 +1,7 @@
 name: CLI Tests
 
 on:
+  workflow_dispatch:
   push:
     branches: [main]
     paths:

.github/workflows/pr-ready-review.yml

@@ -22,6 +22,15 @@ jobs:
         with:
           fetch-depth: 1
 
+      - name: Read review prompt
+        id: review-prompt
+        run: |
+          {
+            echo 'REVIEW_PROMPT<<EOF'
+            cat .claude/review-prompt.md
+            echo 'EOF'
+          } >> "$GITHUB_ENV"
+
       - name: Automatic PR Review
         uses: anthropics/claude-code-action@v1
         with:
@@ -31,18 +40,7 @@ jobs:
             REPO: ${{ github.repository }}
             PR NUMBER: ${{ github.event.pull_request.number }}
 
-            Please review this pull request and provide comprehensive feedback.
-
-            Focus on:
-            - Code quality and best practices
-            - Potential bugs or issues
-            - Performance considerations
-            - Security implications
-
-            Provide detailed feedback using inline comments for specific issues.
-            Use top-level comments for general observations or praise.
-
-            At the end of your review, add complete instructions to reproduce the added changes through the app interface. These instructions will be given to a tester so he can verify the changes. It should be a short descriptive text (not a step by step or a list) on how to navigate the app (what page, what action, what input, etc) to see the changes.
+            ${{ env.REVIEW_PROMPT }}
           claude_args: |
             --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
             --model opus

CHANGELOG.md

@@ -1,5 +1,118 @@
 # Changelog
 
+## [1.669.1](https://github.com/windmill-labs/windmill/compare/v1.669.0...v1.669.1) (2026-03-30)
+
+
+### Bug Fixes
+
+* avoid doubled /oauth2 path in Okta custom authorization server URLs ([#8620](https://github.com/windmill-labs/windmill/issues/8620)) ([4817913](https://github.com/windmill-labs/windmill/commit/4817913f0cab49980bfeb442089631d7953955ff))
+* improve db health UI text and prevent label wrapping ([d532c1d](https://github.com/windmill-labs/windmill/commit/d532c1d470fcb0ef02ebc5342ad1cf22e58b1f4d))
+
+## [1.669.0](https://github.com/windmill-labs/windmill/compare/v1.668.5...v1.669.0) (2026-03-30)
+
+
+### Features
+
+* WAC workflow diagram visualization via WASM ([#8604](https://github.com/windmill-labs/windmill/issues/8604)) ([abc6b12](https://github.com/windmill-labs/windmill/commit/abc6b12d6815edc4dda3ddf5f0572ecedcb670dd))
+
+
+### Bug Fixes
+
+* add path traversal check in service_logs get_log_file endpoint ([#8605](https://github.com/windmill-labs/windmill/issues/8605)) ([5f2d3e6](https://github.com/windmill-labs/windmill/commit/5f2d3e6812f01fe6194bcfd976970a6e3c4186cc))
+* cast DuckDB IS_NULLABLE to string in metadata query ([#8607](https://github.com/windmill-labs/windmill/issues/8607)) ([f3012ee](https://github.com/windmill-labs/windmill/commit/f3012ee7ccc7a8947b5f6bd7c7df77984437f91e))
+* enable S3 bundle cache for PHP previews without lock file ([#8608](https://github.com/windmill-labs/windmill/issues/8608)) ([ee62315](https://github.com/windmill-labs/windmill/commit/ee6231590ed91063f104e6d054b52e88b569986f))
+* enforce workspace isolation on flow resume endpoint ([#8612](https://github.com/windmill-labs/windmill/issues/8612)) ([33032ed](https://github.com/windmill-labs/windmill/commit/33032ed297cf9ea867388d4ea2ece607c9d36dc7))
+* handle DuckDB boolean types in ColumnDef deserializers ([#8610](https://github.com/windmill-labs/windmill/issues/8610)) ([22da5bd](https://github.com/windmill-labs/windmill/commit/22da5bd9ea1ca000cfab3eecf1e3fb0fc01200cb))
+* use route_service instead of fallback_service for MCP router ([#8614](https://github.com/windmill-labs/windmill/issues/8614)) ([98934d5](https://github.com/windmill-labs/windmill/commit/98934d59c552325fcf88c016e31ae977970e8c9a))
+
+## [1.668.5](https://github.com/windmill-labs/windmill/compare/v1.668.4...v1.668.5) (2026-03-29)
+
+
+### Bug Fixes
+
+* add per-IP and per-account brute force protection on login endpoint ([#8601](https://github.com/windmill-labs/windmill/issues/8601)) ([06bbe7b](https://github.com/windmill-labs/windmill/commit/06bbe7b94bfb846bd73aaf6abdc83e4c14e70adc))
+* add timestamp validation to webhook signature verification ([#8596](https://github.com/windmill-labs/windmill/issues/8596)) ([74fba2a](https://github.com/windmill-labs/windmill/commit/74fba2abf3dc68b682777c01da360258786fded8))
+* disable workspace webhook events when CLOUD_HOSTED ([#8598](https://github.com/windmill-labs/windmill/issues/8598)) ([be7fbeb](https://github.com/windmill-labs/windmill/commit/be7fbeb8b1f31d15e33b0783b2a504d6a01e532e))
+* harden login rate limiting with CLOUD_HOSTED gating and memory eviction ([#8602](https://github.com/windmill-labs/windmill/issues/8602)) ([754b88a](https://github.com/windmill-labs/windmill/commit/754b88a52c4e76421cb21c1eed87ad9d8385e9aa))
+* prevent SSRF and local file read via git repository resource URLs ([#8600](https://github.com/windmill-labs/windmill/issues/8600)) ([845db72](https://github.com/windmill-labs/windmill/commit/845db72b7344fb87ac9c5e24697750549665c7bf))
+* rename snippet param to avoid svelte compiler shadowing bug in asset usages drawer ([#8595](https://github.com/windmill-labs/windmill/issues/8595)) ([8c770a2](https://github.com/windmill-labs/windmill/commit/8c770a206a3b0704642c0bda2ab2aeb199d8af3f))
+* require mcp: scope for MCP endpoints instead of blanket bypass ([#8597](https://github.com/windmill-labs/windmill/issues/8597)) ([f5fc9f8](https://github.com/windmill-labs/windmill/commit/f5fc9f8485d2ec3e20f8b451305195446b90e5a3))
+* use constant-time comparison for API key and basic auth validation ([#8593](https://github.com/windmill-labs/windmill/issues/8593)) ([b4d1f2a](https://github.com/windmill-labs/windmill/commit/b4d1f2aac789306c2e35e123ac93e12c47c26f99))
+* validate JSON before sql_builder bind to prevent injection via JSONB queries ([#8599](https://github.com/windmill-labs/windmill/issues/8599)) ([970e859](https://github.com/windmill-labs/windmill/commit/970e859a410b0144847a1a30d7059955effdd402))
+
+## [1.668.4](https://github.com/windmill-labs/windmill/compare/v1.668.3...v1.668.4) (2026-03-29)
+
+
+### Bug Fixes
+
+* update git sync version to latest cli ([0549f68](https://github.com/windmill-labs/windmill/commit/0549f682fe14f4d4b2f67941362ed2cc29d974a1))
+
+## [1.668.3](https://github.com/windmill-labs/windmill/compare/v1.668.2...v1.668.3) (2026-03-28)
+
+
+### Bug Fixes
+
+* **cli:** phantom diffs, flow safety, trigger DX, lint watch, error clarity ([#8588](https://github.com/windmill-labs/windmill/issues/8588)) ([c6ce319](https://github.com/windmill-labs/windmill/commit/c6ce3197a72ceeffd702cf2263b1074ecbf1ca33))
+
+## [1.668.2](https://github.com/windmill-labs/windmill/compare/v1.668.1...v1.668.2) (2026-03-28)
+
+
+### Bug Fixes
+
+* **cli:** app push crash, lint path, push --message, run validation, history timestamps ([#8585](https://github.com/windmill-labs/windmill/issues/8585)) ([f40cdaf](https://github.com/windmill-labs/windmill/commit/f40cdaf43453d2643800ed730d6abe6873bbe8e7))
+
+## [1.668.1](https://github.com/windmill-labs/windmill/compare/v1.668.0...v1.668.1) (2026-03-28)
+
+
+### Bug Fixes
+
+* **cli:** fix 13 CLI bugs — exit codes, sync tar fallback, variable encryption, JSON output ([#8582](https://github.com/windmill-labs/windmill/issues/8582)) ([38acaa3](https://github.com/windmill-labs/windmill/commit/38acaa3653728bf9e0ae6f746edf433703b4ab63))
+
+## [1.668.0](https://github.com/windmill-labs/windmill/compare/v1.667.0...v1.668.0) (2026-03-28)
+
+
+### Features
+
+* add DB health diagnostic dashboard for superadmins ([#8574](https://github.com/windmill-labs/windmill/issues/8574)) ([9ceab73](https://github.com/windmill-labs/windmill/commit/9ceab730d7def09c2b46527f8a586789d14f2ce0))
+* **cli:** add job, group, audit, token commands and schedule enable/disable ([#8581](https://github.com/windmill-labs/windmill/issues/8581)) ([d29cb23](https://github.com/windmill-labs/windmill/commit/d29cb234dbff07473b911e5e75e362def8a47650))
+* IAM RDS auth for PostgreSQL worker resources ([#8573](https://github.com/windmill-labs/windmill/issues/8573)) ([56253c0](https://github.com/windmill-labs/windmill/commit/56253c04cb679c58d00750da699a6cb62ed52aca))
+
+
+### Bug Fixes
+
+* add Authority Key Identifier to MITM proxy leaf certs ([#8576](https://github.com/windmill-labs/windmill/issues/8576)) ([ce2e6c8](https://github.com/windmill-labs/windmill/commit/ce2e6c8c015110d0385e6afecdc8313aabca1364))
+* Improve CLI developer experience: error handling, sync workflow, JSON output, workspace forks ([#8578](https://github.com/windmill-labs/windmill/issues/8578)) ([501a4ff](https://github.com/windmill-labs/windmill/commit/501a4ff2a94510145952686d24ccc639781beefe))
+* trigger capture filter and focus issues ([#8579](https://github.com/windmill-labs/windmill/issues/8579)) ([820f28f](https://github.com/windmill-labs/windmill/commit/820f28f8799f8dad5cfab94b51ac9921d664f04a))
+
+## [1.667.0](https://github.com/windmill-labs/windmill/compare/v1.666.0...v1.667.0) (2026-03-27)
+
+
+### Features
+
+* add schedule support to CLI branch-specific items ([#8570](https://github.com/windmill-labs/windmill/issues/8570)) ([b592996](https://github.com/windmill-labs/windmill/commit/b592996eee98ddb664f1b007b95a2096d5d4e3a6))
+* add workspace-level service accounts ([#8560](https://github.com/windmill-labs/windmill/issues/8560)) ([3959fe8](https://github.com/windmill-labs/windmill/commit/3959fe82974f5f0383e94fd83a5d78fe4212d56a))
+* **cli:** generate commented wmill.yaml and add config reference command ([#8546](https://github.com/windmill-labs/windmill/issues/8546)) ([d06b426](https://github.com/windmill-labs/windmill/commit/d06b42613f73c4a7b31c990be22b0c97efab2666))
+* DB-coordinated graceful restart staggering for settings changes ([#8555](https://github.com/windmill-labs/windmill/issues/8555)) ([2f32675](https://github.com/windmill-labs/windmill/commit/2f326758013dd1f1e6ae732e5784a32f1fb6e4bd))
+* improve-replay-ui ([#8250](https://github.com/windmill-labs/windmill/issues/8250)) ([c0aafee](https://github.com/windmill-labs/windmill/commit/c0aafee9a9923d5dc2fa3b99da4378e923933a06))
+* support multiple folder selection in MCP scope selector ([#8557](https://github.com/windmill-labs/windmill/issues/8557)) ([ad19ac9](https://github.com/windmill-labs/windmill/commit/ad19ac9b37b04591c921f93f180bdda961af6cef))
+
+
+### Bug Fixes
+
+* **cli:** preserve inline script files during flow generate-locks ([#8561](https://github.com/windmill-labs/windmill/issues/8561)) ([a8b651d](https://github.com/windmill-labs/windmill/commit/a8b651da9ff86766119e14c0b61652be8a7b453a))
+* emit 0 for OTEL queue metrics when tag queue is empty ([#8559](https://github.com/windmill-labs/windmill/issues/8559)) ([79cc4a9](https://github.com/windmill-labs/windmill/commit/79cc4a92d88486c999799826bd0c9663767103f5))
+* handle inline script deletion in sync push + flow new nonDottedPaths ([#8553](https://github.com/windmill-labs/windmill/issues/8553)) ([943fe9c](https://github.com/windmill-labs/windmill/commit/943fe9c6cc9b046e24007e45b5c37afc4804256a))
+* include importer_kind in dependency debounce key to prevent cross-kind collisions ([#8567](https://github.com/windmill-labs/windmill/issues/8567)) ([bc7007b](https://github.com/windmill-labs/windmill/commit/bc7007bb4265e1f1375c1f0678b74325882a4e92))
+* multi-script dedicated workers race on shared job_dir ([#8551](https://github.com/windmill-labs/windmill/issues/8551)) ([#8569](https://github.com/windmill-labs/windmill/issues/8569)) ([63a3573](https://github.com/windmill-labs/windmill/commit/63a3573951d1f724cc63728ed973d039a5468072))
+* preserve notes on nodes inside collapsed groups ([#8552](https://github.com/windmill-labs/windmill/issues/8552)) ([0fb1153](https://github.com/windmill-labs/windmill/commit/0fb115304afc49812420e9ce24e5048502621059))
+* sanitize flow step summaries for filesystem-safe names ([#8554](https://github.com/windmill-labs/windmill/issues/8554)) ([e15bfbf](https://github.com/windmill-labs/windmill/commit/e15bfbf91ee1517432a6861ebb48e129485006aa))
+* use admin db pool in get_copilot_settings_state ([#8564](https://github.com/windmill-labs/windmill/issues/8564)) ([70f3ee5](https://github.com/windmill-labs/windmill/commit/70f3ee5ed4470e9993be822874f2b38e83a96611))
+
+
+### Performance Improvements
+
+* enable bun bundle caching for WAC v2 scripts ([#8556](https://github.com/windmill-labs/windmill/issues/8556)) ([ab868e9](https://github.com/windmill-labs/windmill/commit/ab868e9ebceadaa55e54770d9d59dc5524da13ff))
+
 ## [1.666.0](https://github.com/windmill-labs/windmill/compare/v1.665.0...v1.666.0) (2026-03-26)
 
 

backend/.sqlx/query-077467cd813d5af161cb1cc232724f26984822d4c28ba36c0a9331273b10edc0.json

@@ -0,0 +1,12 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO email_to_igroup (email, igroup) VALUES ('alice@example.com', 'admins') ON CONFLICT DO NOTHING",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": []
+  },
+  "hash": "077467cd813d5af161cb1cc232724f26984822d4c28ba36c0a9331273b10edc0"
+}

backend/.sqlx/query-07770a002a49428c4f956cfc7262d6b6792ae5b97ed90b0ee07d17480b2dffe2.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT setting::bigint as \"max!\" FROM pg_settings WHERE name = 'max_connections'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "max!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "07770a002a49428c4f956cfc7262d6b6792ae5b97ed90b0ee07d17480b2dffe2"
+}

backend/.sqlx/query-143acebe5d815c5d828013ebe46274f891f953c75f821499552ab7794f75063d.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements') as \"exists!\"",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "exists!",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "143acebe5d815c5d828013ebe46274f891f953c75f821499552ab7794f75063d"
+}

backend/.sqlx/query-1721f8b52ea265c0537fd7c742deddf0afbe5cf0d81b15e487c411ae169d3a89.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT igroup FROM email_to_igroup WHERE email = 'alice@example.com'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "igroup",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "1721f8b52ea265c0537fd7c742deddf0afbe5cf0d81b15e487c411ae169d3a89"
+}

backend/.sqlx/query-1cf8597b9d37ec5a924aff8cbc0a05768ed9a679ba908ab16497a9bd55578ba1.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT usr.*, password.super_admin, password.name FROM usr LEFT JOIN password ON usr.email = password.email Where usr.username = $1 AND workspace_id = $2\n        ",
+  "query": "SELECT usr.*, COALESCE(password.super_admin, false) as \"super_admin!\", password.name FROM usr LEFT JOIN password ON usr.email = password.email Where usr.username = $1 AND workspace_id = $2\n        ",
   "describe": {
     "columns": [
       {
@@ -50,11 +50,16 @@
       },
       {
         "ordinal": 9,
-        "name": "super_admin",
+        "name": "is_service_account",
         "type_info": "Bool"
       },
       {
         "ordinal": 10,
+        "name": "super_admin!",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 11,
         "name": "name",
         "type_info": "Varchar"
       }
@@ -76,8 +81,9 @@
       true,
       true,
       false,
+      null,
       true
     ]
   },
-  "hash": "6aabe704395c9be30c86d15a5d22f3509b4fcea56227b019588837132b64d58b"
+  "hash": "1cf8597b9d37ec5a924aff8cbc0a05768ed9a679ba908ab16497a9bd55578ba1"
 }

backend/.sqlx/query-1dd73eff0e89b84c0316af2760a136afdd19dc34f9f31c4f9de6b0f74bc386a6.json

@@ -0,0 +1,32 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            schemaname || '.' || relname as \"table_name!\",\n            pg_total_relation_size(relid) as \"total_size_bytes!\",\n            pg_size_pretty(pg_total_relation_size(relid)) as \"total_size_pretty!\"\n        FROM pg_catalog.pg_statio_user_tables\n        ORDER BY pg_total_relation_size(relid) DESC\n        LIMIT 15",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "table_name!",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "total_size_bytes!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 2,
+        "name": "total_size_pretty!",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "1dd73eff0e89b84c0316af2760a136afdd19dc34f9f31c4f9de6b0f74bc386a6"
+}

backend/.sqlx/query-250a4e3f1a1f95296f7075bf8780e9c7407e89c8f7636484895e99f5a5e71297.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO usr (workspace_id, username, email, is_admin, operator, added_via)\n             VALUES ($1, 'alice', 'alice@example.com', false, true, $2)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Jsonb"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "250a4e3f1a1f95296f7075bf8780e9c7407e89c8f7636484895e99f5a5e71297"
+}

backend/.sqlx/query-26e62b4509e44a7548957ad4ef217fd46bc03d5dca19344cd3bf7b131fa40ed2.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT value FROM global_settings WHERE name = 'retention_period_secs'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "value",
+        "type_info": "Jsonb"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "26e62b4509e44a7548957ad4ef217fd46bc03d5dca19344cd3bf7b131fa40ed2"
+}

backend/.sqlx/query-2ba03e555d2e09dbd0e2ae5ddfd9a268a675bdb23615c78904cebe7f1e31f400.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO usr (workspace_id, username, email, is_admin, operator, added_via)\n             VALUES ($1, 'alice', 'alice@example.com', true, false, $2)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Jsonb"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "2ba03e555d2e09dbd0e2ae5ddfd9a268a675bdb23615c78904cebe7f1e31f400"
+}

backend/.sqlx/query-2d4ccf3ee19a70cbb5bd034c74703bbb30f217cd3673821e11bae3bf9f925720.json

@@ -0,0 +1,32 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            ws.workspace_id as \"workspace_id!\",\n            dt.key as \"name!\",\n            dt.value->>'table_name' as \"table_name\"\n        FROM workspace_settings ws,\n             jsonb_each(ws.datatable) dt\n        WHERE dt.value->>'resource_type' = 'instance'\n          AND dt.value->>'table_name' IS NOT NULL",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "workspace_id!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "name!",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "table_name",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      null,
+      null
+    ]
+  },
+  "hash": "2d4ccf3ee19a70cbb5bd034c74703bbb30f217cd3673821e11bae3bf9f925720"
+}

backend/.sqlx/query-2d95191e899d60385b32f36f2e38137e4173a34c54344ee522745640d48b8813.json

@@ -0,0 +1,32 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            COUNT(*) as \"total!\",\n            COUNT(*) FILTER (WHERE state = 'active') as \"active!\",\n            COUNT(*) FILTER (WHERE state = 'idle') as \"idle!\"\n        FROM pg_stat_activity\n        WHERE backend_type = 'client backend'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "total!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 1,
+        "name": "active!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 2,
+        "name": "idle!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "2d95191e899d60385b32f36f2e38137e4173a34c54344ee522745640d48b8813"
+}

backend/.sqlx/query-30930bfb0513f1a70194a900011b2e890bc4146bb0419210cd76743cacda8bfa.json

@@ -0,0 +1,44 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            table_name as \"table_name!\",\n            SUM(live_tuples)::bigint as \"live_tuples!\",\n            SUM(dead_tuples)::bigint as \"dead_tuples!\",\n            MAX(last_autovacuum) as \"last_autovacuum\",\n            MAX(last_autoanalyze) as \"last_autoanalyze\"\n        FROM (\n            SELECT\n                CASE\n                    WHEN i.inhparent IS NOT NULL THEN schemaname || '.' || p.relname\n                    ELSE schemaname || '.' || s.relname\n                END as table_name,\n                COALESCE(n_live_tup, 0) as live_tuples,\n                COALESCE(n_dead_tup, 0) as dead_tuples,\n                last_autovacuum,\n                last_autoanalyze\n            FROM pg_stat_user_tables s\n            LEFT JOIN pg_class c ON c.relname = s.relname AND c.relnamespace = (\n                SELECT oid FROM pg_namespace WHERE nspname = s.schemaname\n            )\n            LEFT JOIN pg_inherits i ON i.inhrelid = c.oid\n            LEFT JOIN pg_class p ON p.oid = i.inhparent\n        ) sub\n        GROUP BY table_name\n        ORDER BY SUM(dead_tuples) DESC",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "table_name!",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "live_tuples!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 2,
+        "name": "dead_tuples!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 3,
+        "name": "last_autovacuum",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 4,
+        "name": "last_autoanalyze",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null,
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "30930bfb0513f1a70194a900011b2e890bc4146bb0419210cd76743cacda8bfa"
+}

backend/.sqlx/query-359cd29f531d263a8cf7205e0869229a610767087f01e0154be8da0620fa114b.json

@@ -0,0 +1,38 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "WITH all_audit AS (SELECT username, operation, timestamp FROM audit_partitioned UNION ALL SELECT username, operation, timestamp FROM audit),\n        active_users as (SELECT distinct username as email FROM all_audit WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n        active_authors as (SELECT distinct email FROM usr WHERE usr.operator IS false AND email IN (SELECT email FROM active_users)),\n        active_authors_agg as (SELECT array_agg(email) as authors FROM active_authors),\n        active_ops_agg as (SELECT array_agg(email) as operators from active_users WHERE email NOT IN (SELECT email FROM active_authors))\n        SELECT active_authors_agg.authors, active_ops_agg.operators, array_length(active_authors_agg.authors, 1) as author_count, array_length(active_ops_agg.operators, 1) as operator_count FROM active_authors_agg, active_ops_agg",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "authors",
+        "type_info": "VarcharArray"
+      },
+      {
+        "ordinal": 1,
+        "name": "operators",
+        "type_info": "VarcharArray"
+      },
+      {
+        "ordinal": 2,
+        "name": "author_count",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 3,
+        "name": "operator_count",
+        "type_info": "Int4"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "359cd29f531d263a8cf7205e0869229a610767087f01e0154be8da0620fa114b"
+}

backend/.sqlx/query-384f5e9b2ab8e430141e28ea58854cbcfbcf96fd2adbf0513ce942cfe9bceaf0.json

@@ -0,0 +1,26 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT pg_database_size(current_database()) as size_bytes, pg_size_pretty(pg_database_size(current_database())) as size_pretty",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "size_bytes",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 1,
+        "name": "size_pretty",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null
+    ]
+  },
+  "hash": "384f5e9b2ab8e430141e28ea58854cbcfbcf96fd2adbf0513ce942cfe9bceaf0"
+}

backend/.sqlx/query-3bd4f38a1629a69ddda622b6b436198b47c2fe1a507358d49f05031e7beedab6.json

@@ -0,0 +1,34 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n            SELECT workspace_id,\n                   auto_invite->'instance_groups_roles' as instance_groups_roles,\n                   auto_invite->'instance_groups' as instance_groups_json\n            FROM workspace_settings\n            WHERE auto_invite->'instance_groups' ? $1\n            ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "workspace_id",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "instance_groups_roles",
+        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 2,
+        "name": "instance_groups_json",
+        "type_info": "Jsonb"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      null
+    ]
+  },
+  "hash": "3bd4f38a1629a69ddda622b6b436198b47c2fe1a507358d49f05031e7beedab6"
+}

backend/.sqlx/query-544a02447bb2cbe8354a5c4ae93685848af38a3461257a9734c43cbd7bd905cb.json

@@ -0,0 +1,35 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT email, is_service_account, disabled FROM usr WHERE username = $1 AND workspace_id = $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "email",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "is_service_account",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 2,
+        "name": "disabled",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "544a02447bb2cbe8354a5c4ae93685848af38a3461257a9734c43cbd7bd905cb"
+}

backend/.sqlx/query-5d160ba4958583f1ad42de846c544d8d8e81e1b54925a0c5f2cedc1817d99a1b.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT igroup FROM email_to_igroup WHERE email = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "igroup",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "5d160ba4958583f1ad42de846c544d8d8e81e1b54925a0c5f2cedc1817d99a1b"
+}

backend/.sqlx/query-5d6adbe21b9f8dd984d1bfc750fb81763d8650c1316bb0b20816f1a5d61a678c.json

@@ -47,6 +47,11 @@
         "ordinal": 8,
         "name": "added_via",
         "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 9,
+        "name": "is_service_account",
+        "type_info": "Bool"
       }
     ],
     "parameters": {
@@ -63,7 +68,8 @@
       false,
       false,
       true,
-      true
+      true,
+      false
     ]
   },
   "hash": "5d6adbe21b9f8dd984d1bfc750fb81763d8650c1316bb0b20816f1a5d61a678c"

backend/.sqlx/query-60b3a59805d463a61eed68072d1ea032b00fc9bd7a6db22f530f67eb9730fa3b.json

@@ -47,6 +47,11 @@
         "ordinal": 8,
         "name": "added_via",
         "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 9,
+        "name": "is_service_account",
+        "type_info": "Bool"
       }
     ],
     "parameters": {
@@ -64,7 +69,8 @@
       false,
       false,
       true,
-      true
+      true,
+      false
     ]
   },
   "hash": "60b3a59805d463a61eed68072d1ea032b00fc9bd7a6db22f530f67eb9730fa3b"

backend/.sqlx/query-62e8e443cf063fcb30799d9c8971c00d761d54811936deb87a0315ca9cdc9769.json

@@ -0,0 +1,32 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT is_admin, operator, added_via FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "is_admin",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 1,
+        "name": "operator",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 2,
+        "name": "added_via",
+        "type_info": "Jsonb"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      false,
+      true
+    ]
+  },
+  "hash": "62e8e443cf063fcb30799d9c8971c00d761d54811936deb87a0315ca9cdc9769"
+}

backend/.sqlx/query-66e2f8468ba64f22b7a7caa18639d7c833ac2ec573bd89d878b5c8b1afc74d3a.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n        SELECT\n            workspace_id,\n            auto_invite->'instance_groups_roles' as instance_groups_roles\n        FROM workspace_settings\n        WHERE\n            auto_invite->'instance_groups' IS NOT NULL\n            AND auto_invite->'instance_groups' ? $1\n        ",
+  "query": "\n        SELECT\n            workspace_id,\n            auto_invite->'instance_groups_roles' as instance_groups_roles,\n            auto_invite->'instance_groups' as instance_groups_json\n        FROM workspace_settings\n        WHERE\n            auto_invite->'instance_groups' IS NOT NULL\n            AND auto_invite->'instance_groups' ? $1\n        ",
   "describe": {
     "columns": [
       {
@@ -12,6 +12,11 @@
         "ordinal": 1,
         "name": "instance_groups_roles",
         "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 2,
+        "name": "instance_groups_json",
+        "type_info": "Jsonb"
       }
     ],
     "parameters": {
@@ -21,8 +26,9 @@
     },
     "nullable": [
       false,
+      null,
       null
     ]
   },
-  "hash": "7e01ef5799168c0fc2779d42ce352827e2fda6711c0a1b104ca6435ddb14b47d"
+  "hash": "66e2f8468ba64f22b7a7caa18639d7c833ac2ec573bd89d878b5c8b1afc74d3a"
 }

backend/.sqlx/query-68c19cb0e18b94870bbe81f9aab92ba37da67cd2a56834c9d1378eab7551284d.json

@@ -0,0 +1,30 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n            UPDATE kafka_trigger\n            SET\n                kafka_resource_path = $1,\n                group_id = $2,\n                topics = $3,\n                filters = $4,\n                filter_logic = $5,\n                auto_offset_reset = $6,\n                auto_commit = $7,\n                script_path = $8,\n                path = $9,\n                is_flow = $10,\n                edited_by = $11,\n                permissioned_as = $12,\n                edited_at = now(),\n                server_id = NULL,\n                error = NULL,\n                error_handler_path = $15,\n                error_handler_args = $16,\n                retry = $17\n            WHERE\n                workspace_id = $13 AND path = $14\n            ",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "VarcharArray",
+        "JsonbArray",
+        "Varchar",
+        "Varchar",
+        "Bool",
+        "Varchar",
+        "Varchar",
+        "Bool",
+        "Varchar",
+        "Varchar",
+        "Text",
+        "Text",
+        "Varchar",
+        "Jsonb",
+        "Jsonb"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "68c19cb0e18b94870bbe81f9aab92ba37da67cd2a56834c9d1378eab7551284d"
+}

backend/.sqlx/query-68d1370fa02f4fe585684a91e898c4aed45e6b8f409bb33c2681f92265922040.json

@@ -0,0 +1,24 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT EXISTS(SELECT 1 FROM usr WHERE workspace_id = $1 AND (username = $2 OR email = $3))",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "exists",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "68d1370fa02f4fe585684a91e898c4aed45e6b8f409bb33c2681f92265922040"
+}

backend/.sqlx/query-6948eb5aabf82f2f4a08dd4410eb472080ecab3ed652912397245e5216ae0389.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n            INSERT INTO websocket_trigger (\n                workspace_id,\n                path,\n                url,\n                script_path,\n                is_flow,\n                mode,\n                filters,\n                initial_messages,\n                url_runnable_args,\n                edited_by,\n                can_return_message,\n                can_return_error_result,\n                permissioned_as,\n                edited_at,\n                error_handler_path,\n                error_handler_args,\n                retry\n            ) VALUES (\n                $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, now(), $14, $15, $16\n            )\n            ",
+  "query": "\n            INSERT INTO websocket_trigger (\n                workspace_id,\n                path,\n                url,\n                script_path,\n                is_flow,\n                mode,\n                filters,\n                filter_logic,\n                initial_messages,\n                url_runnable_args,\n                edited_by,\n                can_return_message,\n                can_return_error_result,\n                permissioned_as,\n                edited_at,\n                error_handler_path,\n                error_handler_args,\n                retry\n            ) VALUES (\n                $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, now(), $15, $16, $17\n            )\n            ",
   "describe": {
     "columns": [],
     "parameters": {
@@ -23,6 +23,7 @@
           }
         },
         "JsonbArray",
+        "Varchar",
         "JsonbArray",
         "Jsonb",
         "Varchar",
@@ -36,5 +37,5 @@
     },
     "nullable": []
   },
-  "hash": "942c0abb55c910862fd45d3fa56a4eb6729f1a658101bda2d0b0fca96b3cfee5"
+  "hash": "6948eb5aabf82f2f4a08dd4410eb472080ecab3ed652912397245e5216ae0389"
 }

backend/.sqlx/query-6a8f4ed9946bb2a3c5e90695c90b70aa2e83fcb5aa0c953febdd9bac2d95bbec.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n            INSERT INTO kafka_trigger (\n                workspace_id,\n                path,\n                kafka_resource_path,\n                group_id,\n                topics,\n                filters,\n                auto_offset_reset,\n                auto_commit,\n                script_path,\n                is_flow,\n                mode,\n                edited_by,\n                permissioned_as,\n                edited_at,\n                error_handler_path,\n                error_handler_args,\n                retry\n            ) VALUES (\n                $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, now(), $14, $15, $16\n            )\n            ",
+  "query": "\n            INSERT INTO kafka_trigger (\n                workspace_id,\n                path,\n                kafka_resource_path,\n                group_id,\n                topics,\n                filters,\n                filter_logic,\n                auto_offset_reset,\n                auto_commit,\n                script_path,\n                is_flow,\n                mode,\n                edited_by,\n                permissioned_as,\n                edited_at,\n                error_handler_path,\n                error_handler_args,\n                retry\n            ) VALUES (\n                $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, now(), $15, $16, $17\n            )\n            ",
   "describe": {
     "columns": [],
     "parameters": {
@@ -12,6 +12,7 @@
         "VarcharArray",
         "JsonbArray",
         "Varchar",
+        "Varchar",
         "Bool",
         "Varchar",
         "Bool",
@@ -36,5 +37,5 @@
     },
     "nullable": []
   },
-  "hash": "a0a545fda5f3ebea0113d5daaf13358c964d9fb0f41bf2a1c834305b4d2398f2"
+  "hash": "6a8f4ed9946bb2a3c5e90695c90b70aa2e83fcb5aa0c953febdd9bac2d95bbec"
 }

backend/.sqlx/query-6f941e4454f736b32eaef80cdfb9582d6e75af3dc159e7c5f12497d3957f1eef.json

@@ -0,0 +1,12 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO usr (workspace_id, username, email, is_admin, operator)\n             VALUES ('ws-multi-group', 'alice', 'alice@example.com', true, false)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": []
+  },
+  "hash": "6f941e4454f736b32eaef80cdfb9582d6e75af3dc159e7c5f12497d3957f1eef"
+}

backend/.sqlx/query-7c5db0b3bd1dd1f766e1841ca620871a468033e05b6e0188ea4775b63fc66e84.json

@@ -0,0 +1,40 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            c.relname as \"table_name!\",\n            pg_total_relation_size(c.oid) as \"size_bytes!\",\n            pg_size_pretty(pg_total_relation_size(c.oid)) as \"size_pretty!\",\n            COALESCE(c.reltuples, 0) as \"estimated_rows!\"\n        FROM pg_class c\n        JOIN pg_namespace n ON n.oid = c.relnamespace\n        WHERE n.nspname = 'public' AND c.relname = ANY($1)",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "table_name!",
+        "type_info": "Name"
+      },
+      {
+        "ordinal": 1,
+        "name": "size_bytes!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 2,
+        "name": "size_pretty!",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 3,
+        "name": "estimated_rows!",
+        "type_info": "Float4"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "NameArray"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "7c5db0b3bd1dd1f766e1841ca620871a468033e05b6e0188ea4775b63fc66e84"
+}

backend/.sqlx/query-88a467f3c943b134a81ac69c3c6686d1ce1ff2f5aafc15ff1b63cfa86c09c4f0.json

@@ -0,0 +1,26 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT is_admin, operator FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "is_admin",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 1,
+        "name": "operator",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      false
+    ]
+  },
+  "hash": "88a467f3c943b134a81ac69c3c6686d1ce1ff2f5aafc15ff1b63cfa86c09c4f0"
+}

backend/.sqlx/query-9a1483a81f5b086e0765d3d69483e29b09f66090e1f9d394564c16d921d2e66c.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg \n        FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n        WHERE s.workspace_id = $1 AND s.path = $2\n        ORDER by s.created_at DESC",
+  "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg, s.created_at as created_at\n        FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n        WHERE s.workspace_id = $1 AND s.path = $2\n        ORDER by s.created_at DESC",
   "describe": {
     "columns": [
       {
@@ -12,6 +12,11 @@
         "ordinal": 1,
         "name": "deployment_msg",
         "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "created_at",
+        "type_info": "Timestamptz"
       }
     ],
     "parameters": {
@@ -22,8 +27,9 @@
     },
     "nullable": [
       false,
-      true
+      true,
+      false
     ]
   },
-  "hash": "726e956cfcd3ac7c07abeecdf92cf0996efe7fa7b671ac2b3b000ead0ea307de"
+  "hash": "9a1483a81f5b086e0765d3d69483e29b09f66090e1f9d394564c16d921d2e66c"
 }

backend/.sqlx/query-9f07510019ebe6f0c5fa17bf31c2d14755474cba82b3b388a47585a8bb325b1a.json

@@ -0,0 +1,17 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO deployment_metadata (workspace_id, path, script_hash, deployment_msg) VALUES ($1, $2, $3, $4) ON CONFLICT (workspace_id, script_hash) WHERE script_hash IS NOT NULL DO UPDATE SET deployment_msg = EXCLUDED.deployment_msg",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "9f07510019ebe6f0c5fa17bf31c2d14755474cba82b3b388a47585a8bb325b1a"
+}

backend/.sqlx/query-a37c2c4d5656d4b44433de84c454046f7586e36b7bd6a4679d70c359d4aacfcf.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT COUNT(*) FROM usr WHERE is_service_account = true",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "count",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "a37c2c4d5656d4b44433de84c454046f7586e36b7bd6a4679d70c359d4aacfcf"
+}

backend/.sqlx/query-a37cfc632dd37cf37c06743239b5ebc784e5da5ee25d47af187a75220d8fded7.json

@@ -1,29 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "\n            UPDATE kafka_trigger\n            SET\n                kafka_resource_path = $1,\n                group_id = $2,\n                topics = $3,\n                filters = $4,\n                auto_offset_reset = $5,\n                auto_commit = $6,\n                script_path = $7,\n                path = $8,\n                is_flow = $9,\n                edited_by = $10,\n                permissioned_as = $11,\n                edited_at = now(),\n                server_id = NULL,\n                error = NULL,\n                error_handler_path = $14,\n                error_handler_args = $15,\n                retry = $16\n            WHERE\n                workspace_id = $12 AND path = $13\n            ",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Varchar",
-        "VarcharArray",
-        "JsonbArray",
-        "Varchar",
-        "Bool",
-        "Varchar",
-        "Varchar",
-        "Bool",
-        "Varchar",
-        "Varchar",
-        "Text",
-        "Text",
-        "Varchar",
-        "Jsonb",
-        "Jsonb"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "a37cfc632dd37cf37c06743239b5ebc784e5da5ee25d47af187a75220d8fded7"
-}

backend/.sqlx/query-a860dd9722f608184c4b1ef5e609b20cd61f9967a2012fc1c8fe352ee7596358.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO pip_resolution_cache (hash, lockfile, expiration) VALUES ($1, $2, now() + ('7 days')::interval) ON CONFLICT (hash) DO UPDATE SET lockfile = EXCLUDED.lockfile, expiration = EXCLUDED.expiration",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "a860dd9722f608184c4b1ef5e609b20cd61f9967a2012fc1c8fe352ee7596358"
+}

backend/.sqlx/query-a9c3461ca3053f699c957f61780d1e889ad53dc5bf1669c24c0666c290656c00.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT AVG(pg_column_size(result))::bigint as \"avg_size\"\n        FROM (\n            SELECT result FROM v2_job_completed\n            WHERE completed_at > now() - interval '30 days'\n              AND result IS NOT NULL\n            ORDER BY completed_at DESC\n            LIMIT $1\n        ) sub",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "avg_size",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "a9c3461ca3053f699c957f61780d1e889ad53dc5bf1669c24c0666c290656c00"
+}

backend/.sqlx/query-add01e9e31d64e88b84c9505fe3de553031e581b1bb173413a9a3e3eb0817b43.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO usr_to_group (workspace_id, usr, group_) VALUES ($1, $2, $3)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "Varchar"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "add01e9e31d64e88b84c9505fe3de553031e581b1bb173413a9a3e3eb0817b43"
+}

backend/.sqlx/query-b38bd869477a729279cac3ccd4825191fb49e17e5f7e7297c0c819f52b486f49.json

@@ -0,0 +1,18 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE usr SET is_admin = $1, operator = $2, added_via = $3 WHERE workspace_id = $4 AND email = $5 AND added_via->>'source' = 'instance_group'",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Bool",
+        "Bool",
+        "Jsonb",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "b38bd869477a729279cac3ccd4825191fb49e17e5f7e7297c0c819f52b486f49"
+}

backend/.sqlx/query-b760be4a0a80853073a061f7c9ebc2d411294d57b07d54d15d178db3c6ee2a30.json

@@ -0,0 +1,26 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT MIN(completed_at) as oldest, COUNT(*) as total FROM v2_job_completed",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "oldest",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 1,
+        "name": "total",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null
+    ]
+  },
+  "hash": "b760be4a0a80853073a061f7c9ebc2d411294d57b07d54d15d178db3c6ee2a30"
+}

backend/.sqlx/query-c73e98e5a937f44724a96ee1b74d31fa71a7be3b8ba3dec9f59f54a6c4030462.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg \n        FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n        WHERE s.workspace_id = $1 AND s.path = $2\n        ORDER by s.created_at DESC LIMIT 1",
+  "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg, s.created_at as created_at\n        FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n        WHERE s.workspace_id = $1 AND s.path = $2\n        ORDER by s.created_at DESC LIMIT 1",
   "describe": {
     "columns": [
       {
@@ -12,6 +12,11 @@
         "ordinal": 1,
         "name": "deployment_msg",
         "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "created_at",
+        "type_info": "Timestamptz"
       }
     ],
     "parameters": {
@@ -22,8 +27,9 @@
     },
     "nullable": [
       false,
-      true
+      true,
+      false
     ]
   },
-  "hash": "cf2a6ad6471a40b6298775cda9300aeecdd75503bed59d80cd62091d1642d1ec"
+  "hash": "c73e98e5a937f44724a96ee1b74d31fa71a7be3b8ba3dec9f59f54a6c4030462"
 }

backend/.sqlx/query-c7aed7fe3b6774477d403bc3e7fcbce7cdbdd1feb553718cbde60bb8ccff4733.json

@@ -1,29 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "\n        UPDATE\n            websocket_trigger\n        SET\n            url = $1,\n            script_path = $2,\n            path = $3,\n            is_flow = $4,\n            filters = $5,\n            initial_messages = $6,\n            url_runnable_args = $7,\n            edited_by = $8,\n            permissioned_as = $9,\n            can_return_message = $10,\n            can_return_error_result = $11,\n            edited_at = now(),\n            server_id = NULL,\n            error = NULL,\n            error_handler_path = $14,\n            error_handler_args = $15,\n            retry = $16\n        WHERE\n            workspace_id = $12 AND path = $13\n    ",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Varchar",
-        "Varchar",
-        "Bool",
-        "JsonbArray",
-        "JsonbArray",
-        "Jsonb",
-        "Varchar",
-        "Varchar",
-        "Bool",
-        "Bool",
-        "Text",
-        "Text",
-        "Varchar",
-        "Jsonb",
-        "Jsonb"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "c7aed7fe3b6774477d403bc3e7fcbce7cdbdd1feb553718cbde60bb8ccff4733"
-}

backend/.sqlx/query-cb3862634f18160207ee2621ddfca43f00456a27fda32583846497116f92f96c.json

@@ -1,38 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "WITH active_users as (SELECT distinct username as email FROM audit WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n        active_authors as (SELECT distinct email FROM usr WHERE usr.operator IS false AND email IN (SELECT email FROM active_users)),\n        active_authors_agg as (SELECT array_agg(email) as authors FROM active_authors),\n        active_ops_agg as (SELECT array_agg(email) as operators from active_users WHERE email NOT IN (SELECT email FROM active_authors))\n        SELECT active_authors_agg.authors, active_ops_agg.operators, array_length(active_authors_agg.authors, 1) as author_count, array_length(active_ops_agg.operators, 1) as operator_count FROM active_authors_agg, active_ops_agg",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "authors",
-        "type_info": "VarcharArray"
-      },
-      {
-        "ordinal": 1,
-        "name": "operators",
-        "type_info": "VarcharArray"
-      },
-      {
-        "ordinal": 2,
-        "name": "author_count",
-        "type_info": "Int4"
-      },
-      {
-        "ordinal": 3,
-        "name": "operator_count",
-        "type_info": "Int4"
-      }
-    ],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": [
-      null,
-      null,
-      null,
-      null
-    ]
-  },
-  "hash": "cb3862634f18160207ee2621ddfca43f00456a27fda32583846497116f92f96c"
-}

backend/.sqlx/query-d43a4ff78e48580815fb912c98639a08d45596a9f11a2dcf5b1e0d135844ecda.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT value FROM global_settings WHERE name = 'plain_emails_telemetry'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "value",
+        "type_info": "Jsonb"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "d43a4ff78e48580815fb912c98639a08d45596a9f11a2dcf5b1e0d135844ecda"
+}

backend/.sqlx/query-dbc5924bca3aa0b32e296b73f8a967bed68332caf526216597f10ffa5fa951c7.json

@@ -0,0 +1,46 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            c.id as \"id!\",\n            c.workspace_id as \"workspace_id!\",\n            j.runnable_path as \"runnable_path\",\n            pg_column_size(c.result) as \"result_size_bytes!\",\n            c.completed_at as \"completed_at!\"\n        FROM (\n            SELECT id, workspace_id, result, completed_at\n            FROM v2_job_completed\n            WHERE completed_at > now() - interval '30 days'\n              AND result IS NOT NULL\n            ORDER BY completed_at DESC\n            LIMIT $1\n        ) c\n        LEFT JOIN v2_job j ON j.id = c.id\n        WHERE pg_column_size(c.result) > 1024\n        ORDER BY pg_column_size(c.result) DESC\n        LIMIT 10",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id!",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 1,
+        "name": "workspace_id!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 2,
+        "name": "runnable_path",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 3,
+        "name": "result_size_bytes!",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 4,
+        "name": "completed_at!",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      true,
+      null,
+      false
+    ]
+  },
+  "hash": "dbc5924bca3aa0b32e296b73f8a967bed68332caf526216597f10ffa5fa951c7"
+}

backend/.sqlx/query-e3d4f89ce36337af15d237b543eaca47771b480ff194884f9c947dcaf71d6cf9.json

@@ -0,0 +1,30 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        UPDATE\n            websocket_trigger\n        SET\n            url = $1,\n            script_path = $2,\n            path = $3,\n            is_flow = $4,\n            filters = $5,\n            filter_logic = $6,\n            initial_messages = $7,\n            url_runnable_args = $8,\n            edited_by = $9,\n            permissioned_as = $10,\n            can_return_message = $11,\n            can_return_error_result = $12,\n            edited_at = now(),\n            server_id = NULL,\n            error = NULL,\n            error_handler_path = $15,\n            error_handler_args = $16,\n            retry = $17\n        WHERE\n            workspace_id = $13 AND path = $14\n    ",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "Varchar",
+        "Bool",
+        "JsonbArray",
+        "Varchar",
+        "JsonbArray",
+        "Jsonb",
+        "Varchar",
+        "Varchar",
+        "Bool",
+        "Bool",
+        "Text",
+        "Text",
+        "Varchar",
+        "Jsonb",
+        "Jsonb"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "e3d4f89ce36337af15d237b543eaca47771b480ff194884f9c947dcaf71d6cf9"
+}

backend/.sqlx/query-e58ef252b0d2b81e9cd76f394a396abefd791906ada29dd5a7a9148157635ca5.json

@@ -0,0 +1,26 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n            SELECT auto_invite->'instance_groups_roles' as instance_groups_roles,\n                   auto_invite->'instance_groups' as instance_groups_json\n            FROM workspace_settings WHERE workspace_id = 'ws-multi-group'\n            ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "instance_groups_roles",
+        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 1,
+        "name": "instance_groups_json",
+        "type_info": "Jsonb"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null
+    ]
+  },
+  "hash": "e58ef252b0d2b81e9cd76f394a396abefd791906ada29dd5a7a9148157635ca5"
+}

backend/.sqlx/query-e5fb3531f8bc7ef1f7484524f8c3bc9c48f71a44827ba0d01ac5588dc31082a2.json

@@ -47,6 +47,11 @@
         "ordinal": 8,
         "name": "added_via",
         "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 9,
+        "name": "is_service_account",
+        "type_info": "Bool"
       }
     ],
     "parameters": {
@@ -63,7 +68,8 @@
       false,
       false,
       true,
-      true
+      true,
+      false
     ]
   },
   "hash": "e5fb3531f8bc7ef1f7484524f8c3bc9c48f71a44827ba0d01ac5588dc31082a2"

backend/.sqlx/query-f4ad2cf2438c2ae31e388517d09a2c1a2f63ab88cdbc79ffad96c6f9ffb5764b.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO token\n            (token_hash, token_prefix, token, email, label, expiration, super_admin, owner)\n            VALUES ($1, $2, $3, $4, $5, $6, false, $7)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "Varchar",
+        "Varchar",
+        "Varchar",
+        "Timestamptz",
+        "Varchar"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "f4ad2cf2438c2ae31e388517d09a2c1a2f63ab88cdbc79ffad96c6f9ffb5764b"
+}

backend/.sqlx/query-f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO usr\n            (workspace_id, email, username, is_admin, operator, is_service_account)\n            VALUES ($1, $2, $3, false, true, true)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "Varchar"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853"
+}

backend/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "windmill"
-version = "1.666.0"
+version = "1.669.1"
 authors.workspace = true
 edition.workspace = true
 
@@ -79,10 +79,10 @@ members = [
     "./windmill-test-utils",
     "./windmill-api-integration-tests",
 ]
-exclude = ["./windmill-duckdb-ffi-internal"]
+exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
 
 [workspace.package]
-version = "1.666.0"
+version = "1.669.1"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 edition = "2021"
 
@@ -260,6 +260,8 @@ windmill-dep-map.workspace = true
 windmill-test-utils.workspace = true
 windmill-worker-volumes.workspace = true
 windmill-types.workspace = true
+opentelemetry = { workspace = true }
+opentelemetry_sdk = { workspace = true }
 windmill-trigger.workspace = true
 windmill-trigger-websocket.workspace = true
 windmill-trigger-postgres.workspace = true
@@ -362,7 +364,7 @@ reqwest-middleware = { version = "^0", features = ["json"] }
 
 bitflags = "2.9.4"
 memchr = "2.7.4"
-axum = { version = "^0.7", features = ["multipart", "macros"] }
+axum = { version = "^0.8", features = ["multipart", "macros"] }
 headers = "^0"
 hyper = { version = "^1", features = ["full"] }
 hyper-tls = "^0.6"
@@ -371,7 +373,7 @@ tokio = { version = "=1.46.1", features = ["full", "tracing", "time"] }
 tokio-stream = { version = "0.1.17" }
 tower = "^0"
 tower-http = { version = "^0.6", features = ["trace", "cors", "catch-panic"] }
-tower-cookies = "^0.10"
+tower-cookies = "^0.11"
 #stuck because of swc for now
 serde = "=1.0.220"
 serde_json = { version = "^1", features = ["preserve_order", "raw_value"] }
@@ -386,7 +388,7 @@ tracing = "^0"
 tracing-subscriber = { version = "^0", features = ["env-filter", "json"] }
 tracing-appender = "^0"
 prometheus = { version = "^0", default-features = false }
-cookie = { version = "0.17.0" }
+cookie = { version = "0.18.0" }
 phf = { version = "0.11", features = ["macros"] }
 rust-embed = { version = "^6", features = ["interpolate-folder-path"] }
 mime_guess = "^2"
@@ -415,6 +417,7 @@ time = "^0"
 serde_urlencoded = "^0"
 astral-tokio-tar = "^0.5.6"
 tempfile = "^3"
+x509-parser = "^0.16"
 tokio-util = { version = "=0.7.17", features = ["io"] }
 json-pointer = "^0"
 itertools = "^0.14.0"
@@ -566,18 +569,18 @@ flate2 = "^1"
 http = "^1"
 async-stream = "^0"
 
-opentelemetry = "0.27.0"
-tracing-opentelemetry = "0.28.0"
-opentelemetry_sdk = { version = "0.27.1", features = ["rt-tokio"] }
-opentelemetry-otlp = { version = "0.27.0", features = ["grpc-tonic", "tls"] }
-opentelemetry-appender-tracing = "0.27.0"
-opentelemetry-semantic-conventions = { version = "0.27.0", features = ["semconv_experimental"] }
-opentelemetry-proto = { version = "0.29.0", features = ["with-serde", "gen-tonic"] }
+opentelemetry = "0.30.0"
+tracing-opentelemetry = "0.31.0"
+opentelemetry_sdk = { version = "0.30.0", features = ["rt-tokio", "testing"] }
+opentelemetry-otlp = { version = "0.30.0", features = ["grpc-tonic", "tls"] }
+opentelemetry-appender-tracing = "0.30.0"
+opentelemetry-semantic-conventions = { version = "0.30.0", features = ["semconv_experimental"] }
+opentelemetry-proto = { version = "0.30.0", features = ["with-serde", "gen-tonic"] }
 prost = "0.13"
 
 bollard = "0.18.1"
 
-tonic = { version = "=0.12.3", features = ["tls-native-roots"] }
+tonic = { version = "^0.13", features = ["tls-native-roots"] }
 byteorder = "1.5.0"
 
 tikv-jemallocator = { version = "0.5" }

backend/ee-repo-ref.txt

@@ -1 +1 @@
-6db424512b0d02f86489e85f0026581b7637d6e6
+e08a87450627bef9013498e40ee93a47bedda7ee

backend/migrations/20260326200000_service_accounts.down.sql

@@ -0,0 +1 @@
+ALTER TABLE usr DROP COLUMN is_service_account;

backend/migrations/20260326200000_service_accounts.up.sql

@@ -0,0 +1 @@
+ALTER TABLE usr ADD COLUMN IF NOT EXISTS is_service_account BOOLEAN NOT NULL DEFAULT FALSE;

backend/migrations/20260327000000_email_varchar_255.down.sql

@@ -0,0 +1,2 @@
+ALTER TABLE magic_link ALTER COLUMN email TYPE VARCHAR(50);
+ALTER TABLE schedule ALTER COLUMN email TYPE VARCHAR(50);

backend/migrations/20260327000000_email_varchar_255.up.sql

@@ -0,0 +1,2 @@
+ALTER TABLE magic_link ALTER COLUMN email TYPE VARCHAR(255);
+ALTER TABLE schedule ALTER COLUMN email TYPE VARCHAR(255);

backend/migrations/20260328000000_trigger_filter_logic.down.sql

@@ -0,0 +1,2 @@
+ALTER TABLE kafka_trigger DROP COLUMN filter_logic;
+ALTER TABLE websocket_trigger DROP COLUMN filter_logic;

backend/migrations/20260328000000_trigger_filter_logic.up.sql

@@ -0,0 +1,2 @@
+ALTER TABLE kafka_trigger ADD COLUMN filter_logic VARCHAR(3) NOT NULL DEFAULT 'and';
+ALTER TABLE websocket_trigger ADD COLUMN filter_logic VARCHAR(3) NOT NULL DEFAULT 'and';

backend/parsers/windmill-parser-wac/src/dag.rs

@@ -27,11 +27,15 @@ pub struct DagNode {
 #[serde(tag = "type")]
 pub enum DagNodeType {
     Step { name: String, script: String },
+    InlineStep { name: String },
+    Sleep { seconds: String },
+    WaitForApproval,
     Branch { condition_source: String },
     ParallelStart,
     ParallelEnd,
     LoopStart { iter_source: String },
     LoopEnd,
+    Merge,
     Return,
 }
 

backend/parsers/windmill-parser-wac/src/python.rs

@@ -37,7 +37,8 @@ impl LineIndex {
 /// Maps task function name → optional external path (from `@task(path="...")`)
 type TaskFunctions = HashMap<String, Option<String>>;
 
-/// First pass: scan top-level `@task async def foo(...)` declarations.
+/// First pass: scan top-level `@task async def foo(...)` declarations
+/// and `foo = task_script("path")` / `foo = task_flow("path")` assignments.
 fn collect_task_functions(stmts: &[Stmt]) -> TaskFunctions {
     let mut tasks = HashMap::new();
     for stmt in stmts {
@@ -61,6 +62,30 @@ fn collect_task_functions(stmts: &[Stmt]) -> TaskFunctions {
                 }
             }
         }
+        // foo = task_script("path") or foo = task_flow("path")
+        if let Stmt::Assign(assign) = stmt {
+            if let Expr::Call(call) = assign.value.as_ref() {
+                if let Expr::Name(ExprName { id, .. }) = call.func.as_ref() {
+                    if id.as_str() == "task_script" || id.as_str() == "task_flow" {
+                        // Extract the path from the first positional argument
+                        let path = call.args.first().and_then(|arg| {
+                            if let Expr::Constant(c) = arg {
+                                if let rustpython_parser::ast::Constant::Str(s) = &c.value {
+                                    return Some(s.to_string());
+                                }
+                            }
+                            None
+                        });
+                        // Extract variable name from target
+                        if let Some(Expr::Name(ExprName { id: var_name, .. })) =
+                            assign.targets.first()
+                        {
+                            tasks.insert(var_name.to_string(), path);
+                        }
+                    }
+                }
+            }
+        }
     }
     tasks
 }
@@ -88,8 +113,6 @@ struct WacWalker {
     node_counter: usize,
     line_index: LineIndex,
     task_functions: TaskFunctions,
-    in_try: bool,
-    in_while: bool,
     in_nested_func: bool,
     in_comprehension: bool,
 }
@@ -103,8 +126,6 @@ impl WacWalker {
             node_counter: 0,
             line_index: LineIndex::new(source),
             task_functions,
-            in_try: false,
-            in_while: false,
             in_nested_func: false,
             in_comprehension: false,
         }
@@ -292,6 +313,9 @@ impl WacWalker {
         if self.is_task_fn_call(expr) {
             return true;
         }
+        if Self::is_sdk_call(expr) {
+            return true;
+        }
         match expr {
             Expr::Await(ExprAwait { value, .. }) => self.expr_contains_step(value),
             Expr::Call(call) => {
@@ -307,6 +331,17 @@ impl WacWalker {
         }
     }
 
+    /// Check if expr is a call to a known SDK function (step, sleep, wait_for_approval)
+    fn is_sdk_call(expr: &Expr) -> bool {
+        if let Expr::Call(call) = expr {
+            if let Expr::Name(ExprName { id, .. }) = call.func.as_ref() {
+                let name = id.as_str();
+                return name == "step" || name == "sleep" || name == "wait_for_approval";
+            }
+        }
+        false
+    }
+
     /// Walk a list of statements, returning (first_node_id, last_node_id)
     fn walk_body(&mut self, body: &[Stmt]) -> Option<(String, String)> {
         let mut first_id: Option<String> = None;
@@ -353,13 +388,17 @@ impl WacWalker {
     }
 
     fn walk_expr_stmt(&mut self, expr: &Expr) -> Option<(String, String)> {
-        // await task_fn(...)
+        // await task_fn(...) / await step(...) / await sleep(...) / await wait_for_approval(...)
         if let Expr::Await(ExprAwait { value, .. }) = expr {
             // await task_fn(...)
             if let Expr::Call(call) = value.as_ref() {
                 if self.is_task_fn_call(&Expr::Call(call.clone())) {
                     return self.emit_step(call, expr);
                 }
+                // Check for SDK-level calls: step(), sleep(), wait_for_approval()
+                if let Some(result) = self.try_emit_sdk_call(call, expr) {
+                    return Some(result);
+                }
             }
             // await asyncio.gather(task_fn(...), task_fn(...), ...)
             if Self::is_asyncio_gather_call(value) {
@@ -378,17 +417,69 @@ impl WacWalker {
         None
     }
 
+    /// Try to emit a node for SDK-level calls: step(), sleep(), wait_for_approval()
+    fn try_emit_sdk_call(&mut self, call: &ExprCall, expr: &Expr) -> Option<(String, String)> {
+        let callee_name = match call.func.as_ref() {
+            Expr::Name(ExprName { id, .. }) => Some(id.as_str()),
+            _ => None,
+        }?;
+
+        let line = self.line_of_expr(expr);
+
+        match callee_name {
+            "step" => {
+                // step("name", fn) — extract the name from the first string argument
+                let name = call
+                    .args
+                    .first()
+                    .and_then(|arg| {
+                        if let Expr::Constant(c) = arg {
+                            if let rustpython_parser::ast::Constant::Str(s) = &c.value {
+                                return Some(s.to_string());
+                            }
+                        }
+                        None
+                    })
+                    .unwrap_or_else(|| "step".to_string());
+                let id = self.next_id();
+                let node_id = self.add_node(DagNode {
+                    id: id.clone(),
+                    node_type: DagNodeType::InlineStep { name: name.clone() },
+                    label: name,
+                    line,
+                });
+                Some((node_id.clone(), node_id))
+            }
+            "sleep" => {
+                let seconds = call
+                    .args
+                    .first()
+                    .map(|arg| Self::expr_to_source(arg))
+                    .unwrap_or_else(|| "?".to_string());
+                let id = self.next_id();
+                let node_id = self.add_node(DagNode {
+                    id: id.clone(),
+                    node_type: DagNodeType::Sleep { seconds: seconds.clone() },
+                    label: format!("sleep({seconds})"),
+                    line,
+                });
+                Some((node_id.clone(), node_id))
+            }
+            "wait_for_approval" => {
+                let id = self.next_id();
+                let node_id = self.add_node(DagNode {
+                    id: id.clone(),
+                    node_type: DagNodeType::WaitForApproval,
+                    label: "wait_for_approval".to_string(),
+                    line,
+                });
+                Some((node_id.clone(), node_id))
+            }
+            _ => None,
+        }
+    }
+
     fn emit_step(&mut self, call: &ExprCall, expr: &Expr) -> Option<(String, String)> {
-        if self.in_try {
-            self.errors
-                .push(validation::error_step_in_try(self.line_of_expr(expr)));
-            return None;
-        }
-        if self.in_while {
-            self.errors
-                .push(validation::error_step_in_while(self.line_of_expr(expr)));
-            return None;
-        }
         if self.in_nested_func {
             self.errors.push(validation::error_step_in_nested_function(
                 self.line_of_expr(expr),
@@ -416,17 +507,6 @@ impl WacWalker {
     }
 
     fn emit_parallel(&mut self, gather_call: &ExprCall, expr: &Expr) -> Option<(String, String)> {
-        if self.in_try {
-            self.errors
-                .push(validation::error_step_in_try(self.line_of_expr(expr)));
-            return None;
-        }
-        if self.in_while {
-            self.errors
-                .push(validation::error_step_in_while(self.line_of_expr(expr)));
-            return None;
-        }
-
         let line = self.line_of_expr(expr);
         let start_id = self.next_id();
         let start_node_id = self.add_node(DagNode {
@@ -491,8 +571,6 @@ impl WacWalker {
             line,
         });
 
-        let merge_id = format!("{branch_id}_merge");
-
         let mut last_ids = Vec::new();
 
         if let Some((true_first, true_last)) = self.walk_body(&if_stmt.body) {
@@ -514,7 +592,17 @@ impl WacWalker {
         if last_ids.len() == 1 {
             Some((branch_node_id, last_ids.into_iter().next().unwrap()))
         } else {
-            Some((branch_node_id, merge_id))
+            let merge_id = format!("{branch_id}_merge");
+            let merge_node_id = self.add_node(DagNode {
+                id: merge_id,
+                node_type: DagNodeType::Merge,
+                label: "merge".to_string(),
+                line,
+            });
+            for last in last_ids {
+                self.add_edge(&last, &merge_node_id, None);
+            }
+            Some((branch_node_id, merge_node_id))
         }
     }
 
@@ -552,11 +640,36 @@ impl WacWalker {
     }
 
     fn walk_while(&mut self, while_stmt: &StmtWhile) -> Option<(String, String)> {
-        if self.body_contains_step(&while_stmt.body) {
-            let line = self.line_index.line_of(while_stmt.range.start().to_usize());
-            self.errors.push(validation::error_step_in_while(line));
+        if !self.body_contains_step(&while_stmt.body) {
+            return None;
         }
-        None
+
+        let line = self.line_index.line_of(while_stmt.range.start().to_usize());
+        let condition = Self::expr_to_source(&while_stmt.test);
+
+        let start_id = self.next_id();
+        let start_node_id = self.add_node(DagNode {
+            id: start_id.clone(),
+            node_type: DagNodeType::LoopStart { iter_source: condition },
+            label: "while".to_string(),
+            line,
+        });
+
+        if let Some((body_first, body_last)) = self.walk_body(&while_stmt.body) {
+            self.add_edge(&start_node_id, &body_first, None);
+            self.add_edge(&body_last, &start_node_id, Some("next".to_string()));
+        }
+
+        let end_id = self.next_id();
+        let end_node_id = self.add_node(DagNode {
+            id: end_id.clone(),
+            node_type: DagNodeType::LoopEnd,
+            label: "end while".to_string(),
+            line,
+        });
+        self.add_edge(&start_node_id, &end_node_id, Some("done".to_string()));
+
+        Some((start_node_id, end_node_id))
     }
 
     fn walk_try(&mut self, try_stmt: &StmtTry) -> Option<(String, String)> {
@@ -569,11 +682,17 @@ impl WacWalker {
                 }
             });
 
-        if has_steps {
-            let line = self.line_index.line_of(try_stmt.range.start().to_usize());
-            self.errors.push(validation::error_step_in_try(line));
+        if !has_steps {
+            return None;
         }
-        None
+
+        let line = self.line_index.line_of(try_stmt.range.start().to_usize());
+        self.emit_try_catch_branch(
+            &try_stmt.body,
+            &try_stmt.handlers,
+            &try_stmt.finalbody,
+            line,
+        )
     }
 
     fn walk_try_star(&mut self, try_stmt: &StmtTryStar) -> Option<(String, String)> {
@@ -586,11 +705,81 @@ impl WacWalker {
                 }
             });
 
-        if has_steps {
-            let line = self.line_index.line_of(try_stmt.range.start().to_usize());
-            self.errors.push(validation::error_step_in_try(line));
+        if !has_steps {
+            return None;
         }
-        None
+
+        let line = self.line_index.line_of(try_stmt.range.start().to_usize());
+        self.emit_try_catch_branch(
+            &try_stmt.body,
+            &try_stmt.handlers,
+            &try_stmt.finalbody,
+            line,
+        )
+    }
+
+    fn emit_try_catch_branch(
+        &mut self,
+        try_body: &[Stmt],
+        handlers: &[rustpython_parser::ast::ExceptHandler],
+        finally_body: &[Stmt],
+        line: usize,
+    ) -> Option<(String, String)> {
+        let branch_id = self.next_id();
+        let branch_node_id = self.add_node(DagNode {
+            id: branch_id.clone(),
+            node_type: DagNodeType::Branch { condition_source: "try/except".to_string() },
+            label: "try".to_string(),
+            line,
+        });
+
+        let mut last_ids = Vec::new();
+
+        // Try body
+        if let Some((try_first, try_last)) = self.walk_body(try_body) {
+            self.add_edge(&branch_node_id, &try_first, Some("try".to_string()));
+            last_ids.push(try_last);
+        } else {
+            last_ids.push(branch_node_id.clone());
+        }
+
+        // Except handlers
+        for handler in handlers {
+            match handler {
+                rustpython_parser::ast::ExceptHandler::ExceptHandler(eh) => {
+                    if let Some((catch_first, catch_last)) = self.walk_body(&eh.body) {
+                        self.add_edge(&branch_node_id, &catch_first, Some("except".to_string()));
+                        last_ids.push(catch_last);
+                    }
+                }
+            }
+        }
+
+        // Finally body — sequential after merge
+        let merge_last = if last_ids.len() == 1 {
+            last_ids.into_iter().next().unwrap()
+        } else {
+            let merge_id = format!("{branch_id}_merge");
+            let merge_node_id = self.add_node(DagNode {
+                id: merge_id,
+                node_type: DagNodeType::Merge,
+                label: "merge".to_string(),
+                line,
+            });
+            for last in last_ids {
+                self.add_edge(&last, &merge_node_id, None);
+            }
+            merge_node_id
+        };
+
+        if !finally_body.is_empty() {
+            if let Some((finally_first, finally_last)) = self.walk_body(finally_body) {
+                self.add_edge(&merge_last, &finally_first, None);
+                return Some((branch_node_id, finally_last));
+            }
+        }
+
+        Some((branch_node_id, merge_last))
     }
 
     fn walk_return(&mut self, ret: &StmtReturn) -> Option<(String, String)> {

backend/parsers/windmill-parser-wac/src/typescript.rs

@@ -51,22 +51,29 @@ fn extract_var_name(pat: &Pat) -> Option<String> {
     }
 }
 
-/// Check if expr is `task(async fn)` or `task("path", async fn)`.
-/// Returns Some(optional_path) if it is a task() call.
+/// Check if expr is `task(async fn)`, `task("path", async fn)`,
+/// `taskScript("path")`, or `taskFlow("path")`.
+/// Returns Some(optional_path) if it is a task/taskScript/taskFlow call.
 fn extract_task_call_info(expr: &Expr) -> Option<Option<String>> {
     if let Expr::Call(call) = expr {
         if let Callee::Expr(callee) = &call.callee {
             if let Expr::Ident(ident) = callee.as_ref() {
-                if ident.sym.as_ref() == "task" {
+                let name = ident.sym.as_ref();
+                if name == "task" {
                     // task("f/path", async fn) or task(async fn)
                     if call.args.len() == 2 {
-                        // task("f/path", async fn)
                         let path = extract_string_lit(&call.args[0].expr);
                         return Some(path);
                     } else if call.args.len() == 1 {
-                        // task(async fn)
                         return Some(None);
                     }
+                } else if name == "taskScript" || name == "taskFlow" {
+                    // taskScript("./helper.ts") or taskFlow("f/my_flow")
+                    if let Some(first_arg) = call.args.first() {
+                        let path = extract_string_lit(&first_arg.expr);
+                        return Some(path);
+                    }
+                    return Some(None);
                 }
             }
         }
@@ -81,8 +88,6 @@ struct TsWacWalker {
     node_counter: usize,
     cm: Lrc<SourceMap>,
     task_functions: TaskFunctions,
-    in_try: bool,
-    in_while: bool,
     in_nested_func: bool,
 }
 
@@ -95,8 +100,6 @@ impl TsWacWalker {
             node_counter: 0,
             cm,
             task_functions,
-            in_try: false,
-            in_while: false,
             in_nested_func: false,
         }
     }
@@ -224,6 +227,9 @@ impl TsWacWalker {
         if self.is_task_call(expr) {
             return true;
         }
+        if Self::is_sdk_call(expr) {
+            return true;
+        }
         match expr {
             Expr::Await(await_expr) => self.expr_contains_step(&await_expr.arg),
             Expr::Call(call) => {
@@ -237,6 +243,19 @@ impl TsWacWalker {
         }
     }
 
+    /// Check if expr is a call to a known SDK function (step, sleep, waitForApproval)
+    fn is_sdk_call(expr: &Expr) -> bool {
+        if let Expr::Call(call) = expr {
+            if let Callee::Expr(callee) = &call.callee {
+                if let Expr::Ident(ident) = callee.as_ref() {
+                    let name = ident.sym.as_ref();
+                    return name == "step" || name == "sleep" || name == "waitForApproval";
+                }
+            }
+        }
+        false
+    }
+
     fn walk_body(&mut self, stmts: &[Stmt]) -> Option<(String, String)> {
         let mut first_id: Option<String> = None;
         let mut prev_id: Option<String> = None;
@@ -294,12 +313,16 @@ impl TsWacWalker {
     }
 
     fn walk_expr_stmt(&mut self, expr: &Expr) -> Option<(String, String)> {
-        // await task_fn(...)
+        // await task_fn(...) / await step(...) / await sleep(...) / await waitForApproval(...)
         if let Expr::Await(await_expr) = expr {
             if let Expr::Call(call) = await_expr.arg.as_ref() {
                 if self.is_task_call(&Expr::Call(call.clone())) {
                     return self.emit_step(call, expr);
                 }
+                // Check for SDK-level calls: step(), sleep(), waitForApproval()
+                if let Some(result) = self.try_emit_sdk_call(call, expr) {
+                    return Some(result);
+                }
             }
             // await Promise.all([task_fn(...), ...])
             if Self::is_promise_all(&await_expr.arg) {
@@ -318,17 +341,70 @@ impl TsWacWalker {
         None
     }
 
+    /// Try to emit a node for SDK-level calls: step(), sleep(), waitForApproval()
+    fn try_emit_sdk_call(&mut self, call: &CallExpr, expr: &Expr) -> Option<(String, String)> {
+        let callee_name = match &call.callee {
+            Callee::Expr(callee) => match callee.as_ref() {
+                Expr::Ident(ident) => Some(ident.sym.as_ref().to_string()),
+                _ => None,
+            },
+            _ => None,
+        }?;
+
+        let line = self.span_line(expr.span());
+
+        match callee_name.as_str() {
+            "step" => {
+                // step("name", fn) — extract the name from the first string argument
+                let name = call
+                    .args
+                    .first()
+                    .and_then(|a| extract_string_lit(&a.expr))
+                    .unwrap_or_else(|| "step".to_string());
+                let id = self.next_id();
+                let node_id = self.add_node(DagNode {
+                    id: id.clone(),
+                    node_type: DagNodeType::InlineStep { name: name.clone() },
+                    label: name,
+                    line,
+                });
+                Some((node_id.clone(), node_id))
+            }
+            "sleep" => {
+                // sleep(N) — extract the duration from the first argument
+                let seconds = call
+                    .args
+                    .first()
+                    .map(|a| {
+                        self.cm
+                            .span_to_snippet(a.expr.span())
+                            .unwrap_or_else(|_| "?".to_string())
+                    })
+                    .unwrap_or_else(|| "?".to_string());
+                let id = self.next_id();
+                let node_id = self.add_node(DagNode {
+                    id: id.clone(),
+                    node_type: DagNodeType::Sleep { seconds: seconds.clone() },
+                    label: format!("sleep({seconds})"),
+                    line,
+                });
+                Some((node_id.clone(), node_id))
+            }
+            "waitForApproval" => {
+                let id = self.next_id();
+                let node_id = self.add_node(DagNode {
+                    id: id.clone(),
+                    node_type: DagNodeType::WaitForApproval,
+                    label: "waitForApproval".to_string(),
+                    line,
+                });
+                Some((node_id.clone(), node_id))
+            }
+            _ => None,
+        }
+    }
+
     fn emit_step(&mut self, call: &CallExpr, expr: &Expr) -> Option<(String, String)> {
-        if self.in_try {
-            self.errors
-                .push(validation::error_step_in_catch(self.span_line(expr.span())));
-            return None;
-        }
-        if self.in_while {
-            self.errors
-                .push(validation::error_step_in_while(self.span_line(expr.span())));
-            return None;
-        }
         if self.in_nested_func {
             self.errors.push(validation::error_step_in_nested_function(
                 self.span_line(expr.span()),
@@ -350,17 +426,6 @@ impl TsWacWalker {
     }
 
     fn emit_parallel(&mut self, promise_call: &CallExpr, expr: &Expr) -> Option<(String, String)> {
-        if self.in_try {
-            self.errors
-                .push(validation::error_step_in_catch(self.span_line(expr.span())));
-            return None;
-        }
-        if self.in_while {
-            self.errors
-                .push(validation::error_step_in_while(self.span_line(expr.span())));
-            return None;
-        }
-
         let line = self.span_line(expr.span());
         let start_id = self.next_id();
         let start_node_id = self.add_node(DagNode {
@@ -457,7 +522,16 @@ impl TsWacWalker {
             Some((branch_node_id, last_ids.into_iter().next().unwrap()))
         } else {
             let merge_id = format!("{branch_id}_merge");
-            Some((branch_node_id, merge_id))
+            let merge_node_id = self.add_node(DagNode {
+                id: merge_id,
+                node_type: DagNodeType::Merge,
+                label: "merge".to_string(),
+                line,
+            });
+            for last in last_ids {
+                self.add_edge(&last, &merge_node_id, None);
+            }
+            Some((branch_node_id, merge_node_id))
         }
     }
 
@@ -473,7 +547,7 @@ impl TsWacWalker {
             return None;
         }
         let iter_source = self.expr_to_source(&for_in.right);
-        self.walk_loop_body_with_iter(&for_in.body, for_in.span, &iter_source)
+        self.walk_loop_body_with_iter(&for_in.body, for_in.span, &iter_source, "for")
     }
 
     fn walk_for_of(&mut self, for_of: &ForOfStmt) -> Option<(String, String)> {
@@ -481,7 +555,7 @@ impl TsWacWalker {
             return None;
         }
         let iter_source = self.expr_to_source(&for_of.right);
-        self.walk_loop_body_with_iter(&for_of.body, for_of.span, &iter_source)
+        self.walk_loop_body_with_iter(&for_of.body, for_of.span, &iter_source, "for")
     }
 
     fn walk_loop_body(
@@ -490,7 +564,7 @@ impl TsWacWalker {
         span: swc_common::Span,
         _label: &str,
     ) -> Option<(String, String)> {
-        self.walk_loop_body_with_iter(body, span, "...")
+        self.walk_loop_body_with_iter(body, span, "...", "for")
     }
 
     fn walk_loop_body_with_iter(
@@ -498,13 +572,14 @@ impl TsWacWalker {
         body: &Stmt,
         span: swc_common::Span,
         iter_source: &str,
+        loop_label: &str,
     ) -> Option<(String, String)> {
         let line = self.span_line(span);
         let start_id = self.next_id();
         let start_node_id = self.add_node(DagNode {
             id: start_id.clone(),
             node_type: DagNodeType::LoopStart { iter_source: iter_source.to_string() },
-            label: "for".to_string(),
+            label: loop_label.to_string(),
             line,
         });
 
@@ -526,12 +601,11 @@ impl TsWacWalker {
     }
 
     fn walk_while(&mut self, while_stmt: &WhileStmt) -> Option<(String, String)> {
-        if self.stmt_contains_step(&while_stmt.body) {
-            self.errors.push(validation::error_step_in_while(
-                self.span_line(while_stmt.span),
-            ));
+        if !self.stmt_contains_step(&while_stmt.body) {
+            return None;
         }
-        None
+        let condition = self.expr_to_source(&while_stmt.test);
+        self.walk_loop_body_with_iter(&while_stmt.body, while_stmt.span, &condition, "while")
     }
 
     fn walk_try(&mut self, try_stmt: &TryStmt) -> Option<(String, String)> {
@@ -545,12 +619,62 @@ impl TsWacWalker {
                 .as_ref()
                 .map_or(false, |f| self.body_contains_step(&f.stmts));
 
-        if has_steps {
-            self.errors.push(validation::error_step_in_catch(
-                self.span_line(try_stmt.span),
-            ));
+        if !has_steps {
+            return None;
         }
-        None
+
+        let line = self.span_line(try_stmt.span);
+        let branch_id = self.next_id();
+        let branch_node_id = self.add_node(DagNode {
+            id: branch_id.clone(),
+            node_type: DagNodeType::Branch { condition_source: "try/catch".to_string() },
+            label: "try".to_string(),
+            line,
+        });
+
+        let mut last_ids = Vec::new();
+
+        // Try body
+        if let Some((try_first, try_last)) = self.walk_body(&try_stmt.block.stmts) {
+            self.add_edge(&branch_node_id, &try_first, Some("try".to_string()));
+            last_ids.push(try_last);
+        } else {
+            last_ids.push(branch_node_id.clone());
+        }
+
+        // Catch body
+        if let Some(handler) = &try_stmt.handler {
+            if let Some((catch_first, catch_last)) = self.walk_body(&handler.body.stmts) {
+                self.add_edge(&branch_node_id, &catch_first, Some("catch".to_string()));
+                last_ids.push(catch_last);
+            }
+        }
+
+        // Finally body — sequential after merge
+        let merge_last = if last_ids.len() == 1 {
+            last_ids.into_iter().next().unwrap()
+        } else {
+            let merge_id = format!("{branch_id}_merge");
+            let merge_node_id = self.add_node(DagNode {
+                id: merge_id,
+                node_type: DagNodeType::Merge,
+                label: "merge".to_string(),
+                line,
+            });
+            for last in last_ids {
+                self.add_edge(&last, &merge_node_id, None);
+            }
+            merge_node_id
+        };
+
+        if let Some(finalizer) = &try_stmt.finalizer {
+            if let Some((finally_first, finally_last)) = self.walk_body(&finalizer.stmts) {
+                self.add_edge(&merge_last, &finally_first, None);
+                return Some((branch_node_id, finally_last));
+            }
+        }
+
+        Some((branch_node_id, merge_last))
     }
 
     fn walk_return(&mut self, ret: &ReturnStmt) -> Option<(String, String)> {
@@ -632,6 +756,19 @@ pub fn parse_ts_workflow(code: &str) -> Result<WorkflowDag, Vec<CompileError>> {
                 }
             }
         }
+        // export const main = workflow(async (...) => { ... })
+        if let ModuleItem::ModuleDecl(ModuleDecl::ExportDecl(export)) = item {
+            if let Decl::Var(var_decl) = &export.decl {
+                for decl in &var_decl.decls {
+                    if let Some(init) = &decl.init {
+                        if let Some(result) = find_workflow_call(init, &cm) {
+                            workflow_body = Some(result);
+                            break;
+                        }
+                    }
+                }
+            }
+        }
     }
 
     let (stmts, params) = workflow_body.ok_or_else(|| {

backend/parsers/windmill-parser-wac/src/validation.rs

@@ -12,23 +12,6 @@ impl std::fmt::Display for CompileError {
     }
 }
 
-pub fn error_step_in_try(line: usize) -> CompileError {
-    CompileError {
-        message:
-            "Task calls inside try/except are not allowed. Steps have built-in error handling."
-                .to_string(),
-        line,
-    }
-}
-
-pub fn error_step_in_while(line: usize) -> CompileError {
-    CompileError {
-        message: "Task calls inside while loops are not allowed. Use for loops instead."
-            .to_string(),
-        line,
-    }
-}
-
 pub fn error_step_in_nested_function(line: usize) -> CompileError {
     CompileError {
         message: "Task calls inside nested functions, closures, or lambdas are not allowed."
@@ -53,12 +36,3 @@ pub fn error_missing_await(line: usize) -> CompileError {
         line,
     }
 }
-
-pub fn error_step_in_catch(line: usize) -> CompileError {
-    CompileError {
-        message:
-            "Task calls inside catch blocks are not allowed. Steps have built-in error handling."
-                .to_string(),
-        line,
-    }
-}

backend/parsers/windmill-parser-wac/tests/python_tests.rs

@@ -147,7 +147,7 @@ async def my_etl(items: list):
 }
 
 #[test]
-fn test_reject_step_in_try() {
+fn test_step_in_try_except() {
     let code = r#"
 import asyncio
 from wmill import workflow, task
@@ -155,39 +155,52 @@ from wmill import workflow, task
 @task
 async def extract_data(): ...
 
+@task
+async def handle_error(): ...
+
 @workflow
 async def my_etl():
     try:
         await extract_data()
     except Exception:
-        pass
+        await handle_error()
 "#;
 
-    let result = parse_python_workflow(code);
-    assert!(result.is_err());
-    let errors = result.unwrap_err();
-    assert!(errors[0].message.contains("try/except"));
+    let dag = parse_python_workflow(code).expect("should parse try/except");
+    // Branch(try/except), extract_data, handle_error, merge = 4
+    assert_eq!(dag.nodes.len(), 4);
+    assert!(matches!(dag.nodes[0].node_type, DagNodeType::Branch { .. }));
+    assert_eq!(dag.nodes[0].label, "try");
+    assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. }));
+    assert!(matches!(dag.nodes[2].node_type, DagNodeType::Step { .. }));
+    assert!(matches!(dag.nodes[3].node_type, DagNodeType::Merge));
 }
 
 #[test]
-fn test_reject_step_in_while() {
+fn test_step_in_while() {
     let code = r#"
 import asyncio
 from wmill import workflow, task
 
 @task
-async def extract_data(): ...
+async def poll_status(): ...
 
 @workflow
 async def my_etl():
     while True:
-        await extract_data()
+        await poll_status()
 "#;
 
-    let result = parse_python_workflow(code);
-    assert!(result.is_err());
-    let errors = result.unwrap_err();
-    assert!(errors[0].message.contains("while"));
+    let dag = parse_python_workflow(code).expect("should parse while loop");
+    // LoopStart, poll_status, LoopEnd = 3
+    assert_eq!(dag.nodes.len(), 3);
+    assert!(matches!(
+        dag.nodes[0].node_type,
+        DagNodeType::LoopStart { .. }
+    ));
+    assert_eq!(dag.nodes[0].label, "while");
+    assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. }));
+    assert!(matches!(dag.nodes[2].node_type, DagNodeType::LoopEnd));
 }
 
 #[test]
@@ -264,3 +277,89 @@ async def my_wf(x: int):
         _ => panic!("expected Step node"),
     }
 }
+
+#[test]
+fn test_task_script_and_task_flow_py() {
+    let code = r#"
+from wmill import workflow, task, task_script, task_flow
+
+helper = task_script("./helper.py")
+pipeline = task_flow("f/etl/pipeline")
+
+@task()
+async def process(x: str) -> str:
+    return f"processed: {x}"
+
+@workflow
+async def main(x: str):
+    a = await process(x=x)
+    b = await helper(a=a)
+    c = await pipeline(b=b)
+    return {"a": a, "b": b, "c": c}
+"#;
+
+    let dag = parse_python_workflow(code).expect("should parse");
+    assert_eq!(dag.nodes.len(), 4); // 3 steps + 1 return
+
+    match &dag.nodes[1].node_type {
+        DagNodeType::Step { name, script } => {
+            assert_eq!(name, "helper");
+            assert_eq!(script, "./helper.py");
+        }
+        _ => panic!("expected Step node for task_script"),
+    }
+
+    match &dag.nodes[2].node_type {
+        DagNodeType::Step { name, script } => {
+            assert_eq!(name, "pipeline");
+            assert_eq!(script, "f/etl/pipeline");
+        }
+        _ => panic!("expected Step node for task_flow"),
+    }
+}
+
+#[test]
+fn test_full_template_with_sdk_calls_py() {
+    let code = r#"
+from wmill import workflow, task, task_script, step, sleep, wait_for_approval, get_resume_urls
+
+helper = task_script("./helper.py")
+
+@task()
+async def process(x: str) -> str:
+    return f"processed: {x}"
+
+@workflow
+async def main(x: str):
+    a = await process(x=x)
+    b = await helper(a=a)
+    urls = await step("get_urls", lambda: get_resume_urls())
+    await sleep(1)
+    approval = await wait_for_approval(timeout=3600)
+    return {"processed": a, "helper_result": b, "approval": approval}
+"#;
+
+    let dag = parse_python_workflow(code).expect("should parse");
+    // process, helper, step("get_urls"), sleep(1), wait_for_approval, return = 6
+    assert_eq!(dag.nodes.len(), 6);
+
+    match &dag.nodes[2].node_type {
+        DagNodeType::InlineStep { name } => {
+            assert_eq!(name, "get_urls");
+        }
+        _ => panic!("expected InlineStep node, got {:?}", dag.nodes[2].node_type),
+    }
+
+    match &dag.nodes[3].node_type {
+        DagNodeType::Sleep { seconds } => {
+            assert_eq!(seconds, "1");
+        }
+        _ => panic!("expected Sleep node, got {:?}", dag.nodes[3].node_type),
+    }
+
+    assert!(matches!(
+        dag.nodes[4].node_type,
+        DagNodeType::WaitForApproval
+    ));
+    assert!(matches!(dag.nodes[5].node_type, DagNodeType::Return));
+}

backend/parsers/windmill-parser-wac/tests/ts_tests.rs

@@ -129,45 +129,56 @@ export default workflow(async (items: string[]) => {
 }
 
 #[test]
-fn test_reject_step_in_try_catch() {
+fn test_step_in_try_catch() {
     let code = r#"
 import { workflow, task } from "windmill-client";
 
 const extract_data = task(async () => {});
+const handle_error = task(async (e: any) => {});
 
 export default workflow(async () => {
   try {
     await extract_data();
   } catch (e) {
-    console.log(e);
+    await handle_error(e);
   }
 });
 "#;
 
-    let result = parse_ts_workflow(code);
-    assert!(result.is_err());
-    let errors = result.unwrap_err();
-    assert!(errors[0].message.contains("catch"));
+    let dag = parse_ts_workflow(code).expect("should parse try/catch");
+    // Branch(try/catch), extract_data, handle_error, merge = 4
+    assert_eq!(dag.nodes.len(), 4);
+    assert!(matches!(dag.nodes[0].node_type, DagNodeType::Branch { .. }));
+    assert_eq!(dag.nodes[0].label, "try");
+    assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. }));
+    assert!(matches!(dag.nodes[2].node_type, DagNodeType::Step { .. }));
+    assert!(matches!(dag.nodes[3].node_type, DagNodeType::Merge));
 }
 
 #[test]
-fn test_reject_step_in_while_ts() {
+fn test_step_in_while_ts() {
     let code = r#"
 import { workflow, task } from "windmill-client";
 
-const extract_data = task(async () => {});
+const poll_status = task(async () => {});
 
 export default workflow(async () => {
   while (true) {
-    await extract_data();
+    await poll_status();
   }
 });
 "#;
 
-    let result = parse_ts_workflow(code);
-    assert!(result.is_err());
-    let errors = result.unwrap_err();
-    assert!(errors[0].message.contains("while"));
+    let dag = parse_ts_workflow(code).expect("should parse while loop");
+    // LoopStart, poll_status, LoopEnd = 3
+    assert_eq!(dag.nodes.len(), 3);
+    assert!(matches!(
+        dag.nodes[0].node_type,
+        DagNodeType::LoopStart { .. }
+    ));
+    assert_eq!(dag.nodes[0].label, "while");
+    assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. }));
+    assert!(matches!(dag.nodes[2].node_type, DagNodeType::LoopEnd));
 }
 
 #[test]
@@ -243,3 +254,158 @@ export default workflow(async (x: number) => {
         _ => panic!("expected Step node"),
     }
 }
+
+#[test]
+fn test_task_script_and_task_flow() {
+    let code = r#"
+import { workflow, task, taskScript, taskFlow } from "windmill-client";
+
+const helper = taskScript("./helper.ts");
+const pipeline = taskFlow("f/etl/pipeline");
+const process = task(async (x: string) => {});
+
+export default workflow(async (x: string) => {
+  const a = await process(x);
+  const b = await helper({ a });
+  const c = await pipeline({ b });
+  return { a, b, c };
+});
+"#;
+
+    let dag = parse_ts_workflow(code).expect("should parse");
+    assert_eq!(dag.nodes.len(), 4); // 3 steps + 1 return
+
+    match &dag.nodes[0].node_type {
+        DagNodeType::Step { name, script } => {
+            assert_eq!(name, "process");
+            assert_eq!(script, "process");
+        }
+        _ => panic!("expected Step node"),
+    }
+
+    match &dag.nodes[1].node_type {
+        DagNodeType::Step { name, script } => {
+            assert_eq!(name, "helper");
+            assert_eq!(script, "./helper.ts");
+        }
+        _ => panic!("expected Step node for taskScript"),
+    }
+
+    match &dag.nodes[2].node_type {
+        DagNodeType::Step { name, script } => {
+            assert_eq!(name, "pipeline");
+            assert_eq!(script, "f/etl/pipeline");
+        }
+        _ => panic!("expected Step node for taskFlow"),
+    }
+}
+
+#[test]
+fn test_full_template_with_sdk_calls() {
+    let code = r#"
+import { task, taskScript, step, sleep, waitForApproval, getResumeUrls, workflow } from "windmill-client";
+
+const helper = taskScript("./helper.ts");
+const process = task(async (x: string): Promise<string> => {
+  return `processed: ${x}`;
+});
+
+export const main = workflow(async (x: string) => {
+  const a = await process(x);
+  const b = await helper({ a });
+  const urls = await step("get_urls", () => getResumeUrls());
+  await sleep(1);
+  const approval = await waitForApproval({ timeout: 3600 });
+  return { processed: a, helper_result: b, approval };
+});
+"#;
+
+    let dag = parse_ts_workflow(code).expect("should parse");
+    // process, helper, step("get_urls"), sleep(1), waitForApproval, return = 6
+    assert_eq!(dag.nodes.len(), 6);
+    assert_eq!(dag.edges.len(), 5);
+
+    assert!(matches!(dag.nodes[0].node_type, DagNodeType::Step { .. }));
+
+    match &dag.nodes[1].node_type {
+        DagNodeType::Step { name, script } => {
+            assert_eq!(name, "helper");
+            assert_eq!(script, "./helper.ts");
+        }
+        _ => panic!("expected Step node"),
+    }
+
+    match &dag.nodes[2].node_type {
+        DagNodeType::InlineStep { name } => {
+            assert_eq!(name, "get_urls");
+        }
+        _ => panic!("expected InlineStep node, got {:?}", dag.nodes[2].node_type),
+    }
+
+    match &dag.nodes[3].node_type {
+        DagNodeType::Sleep { seconds } => {
+            assert_eq!(seconds, "1");
+        }
+        _ => panic!("expected Sleep node, got {:?}", dag.nodes[3].node_type),
+    }
+
+    assert!(matches!(
+        dag.nodes[4].node_type,
+        DagNodeType::WaitForApproval
+    ));
+    assert!(matches!(dag.nodes[5].node_type, DagNodeType::Return));
+}
+
+#[test]
+fn test_complex_mixed_workflow() {
+    let code = r#"
+import { workflow, task, step, sleep } from "windmill-client";
+
+const validate = task(async (data: any) => {});
+const process_csv = task(async (data: any) => {});
+const process_json = task(async (data: any) => {});
+const enrich = task(async (item: any) => {});
+const store = task(async (data: any) => {});
+
+export default workflow(async (data: any) => {
+  const validated = await validate(data);
+  if (validated.format === "csv") {
+    const parsed = await process_csv(validated);
+    for (const row of parsed.rows) {
+      await enrich(row);
+    }
+  } else {
+    await process_json(validated);
+  }
+  await sleep(5);
+  const ts = await step("timestamp", () => new Date().toISOString());
+  await store(validated);
+  return { done: true };
+});
+"#;
+
+    let dag = parse_ts_workflow(code).expect("should parse");
+
+    // validate, Branch, process_csv, LoopStart, enrich, LoopEnd, process_json,
+    // merge, sleep(5), step("timestamp"), store, return = 12
+    assert_eq!(dag.nodes.len(), 12);
+
+    assert!(matches!(dag.nodes[0].node_type, DagNodeType::Step { .. }));
+    assert!(matches!(dag.nodes[1].node_type, DagNodeType::Branch { .. }));
+    assert!(matches!(dag.nodes[2].node_type, DagNodeType::Step { .. })); // process_csv
+    assert!(matches!(
+        dag.nodes[3].node_type,
+        DagNodeType::LoopStart { .. }
+    ));
+    assert!(matches!(dag.nodes[4].node_type, DagNodeType::Step { .. })); // enrich
+    assert!(matches!(dag.nodes[5].node_type, DagNodeType::LoopEnd));
+    assert!(matches!(dag.nodes[6].node_type, DagNodeType::Step { .. })); // process_json
+    assert!(matches!(dag.nodes[7].node_type, DagNodeType::Merge));
+    assert!(matches!(dag.nodes[8].node_type, DagNodeType::Sleep { .. }));
+    assert!(matches!(
+        dag.nodes[9].node_type,
+        DagNodeType::InlineStep { .. }
+    )); // timestamp
+    assert!(matches!(dag.nodes[10].node_type, DagNodeType::Step { .. })); // store
+    assert!(matches!(dag.nodes[11].node_type, DagNodeType::Return));
+}

backend/parsers/windmill-parser-wasm/publish-pkgs.sh

@@ -39,3 +39,6 @@ popd
 
 pushd "pkg-py-imports" && npm publish ${args}
 popd
+
+pushd "pkg-wac" && npm publish ${args}
+popd

backend/src/main.rs

@@ -52,9 +52,10 @@ use windmill_common::{
         NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING, OAUTH_SETTING, OTEL_SETTING,
         OTEL_TRACING_PROXY_SETTING, PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING,
         POWERSHELL_REPO_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
-        REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
-        RUBY_REPOS_SETTING, SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, SMTP_SETTING, TEAMS_SETTING,
-        TIMEOUT_WAIT_RESULT_SETTING, UV_INDEX_STRATEGY_SETTING, WORKSPACE_REGISTRIES_SETTING,
+        REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RESTART_COORDINATION_SETTING,
+        RETENTION_PERIOD_SECS_SETTING, RUBY_REPOS_SETTING, SAML_METADATA_SETTING,
+        SCIM_TOKEN_SETTING, SMTP_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
+        UV_INDEX_STRATEGY_SETTING, WORKSPACE_REGISTRIES_SETTING,
     },
     scripts::ScriptLang,
     stats_oss::schedule_stats,
@@ -67,7 +68,7 @@ use windmill_common::{
         is_native_mode_from_env, reload_custom_tags_setting, Connection, HUB_CACHE_DIR,
         HUB_RT_CACHE_DIR, NATIVE_MODE_RESOLVED, TMP_LOGS_DIR, WINDMILL_DIR, WORKER_GROUP,
     },
-    KillpillSender, DEFAULT_HUB_BASE_URL, METRICS_ENABLED,
+    KillpillSender, DEFAULT_HUB_BASE_URL, INSTANCE_NAME, METRICS_ENABLED,
 };
 
 #[cfg(feature = "enterprise")]
@@ -1791,7 +1792,8 @@ async fn process_notify_event(
                     reload_otel_tracing_proxy_setting(conn).await;
                     if worker_mode {
                         tracing::info!("OTEL tracing proxy setting changed, restarting worker");
-                        send_delayed_killpill(tx, 4, "OTEL tracing proxy setting change").await;
+                        spawn_graceful_killpill(tx, db, 10, "OTEL tracing proxy setting change")
+                            .await;
                     }
                 }
                 REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING => {
@@ -1799,12 +1801,12 @@ async fn process_notify_event(
                 }
                 EXPOSE_METRICS_SETTING => {
                     tracing::info!("Metrics setting changed, restarting");
-                    send_delayed_killpill(tx, 40, "metrics setting change").await;
+                    spawn_graceful_killpill(tx, db, 10, "metrics setting change").await;
                 }
                 EMAIL_DOMAIN_SETTING => {
                     tracing::info!("Email domain setting changed");
                     if server_mode {
-                        send_delayed_killpill(tx, 4, "email domain setting change").await;
+                        spawn_graceful_killpill(tx, db, 10, "email domain setting change").await;
                     }
                 }
                 EXPOSE_DEBUG_METRICS_SETTING => {
@@ -1840,19 +1842,19 @@ async fn process_notify_event(
                 }
                 OTEL_SETTING => {
                     tracing::info!("OTEL setting changed, restarting");
-                    send_delayed_killpill(tx, 4, "OTEL setting change").await;
+                    spawn_graceful_killpill(tx, db, 10, "OTEL setting change").await;
                 }
                 REQUEST_SIZE_LIMIT_SETTING => {
                     if server_mode {
                         tracing::info!("Request limit size change detected, killing server expecting to be restarted");
-                        send_delayed_killpill(tx, 4, "request size limit change").await;
+                        spawn_graceful_killpill(tx, db, 10, "request size limit change").await;
                     }
                 }
                 SAML_METADATA_SETTING => {
                     tracing::info!(
                         "SAML metadata change detected, killing server expecting to be restarted"
                     );
-                    send_delayed_killpill(tx, 0, "SAML metadata change").await;
+                    spawn_graceful_killpill(tx, db, 10, "SAML metadata change").await;
                 }
                 HUB_BASE_URL_SETTING => {
                     if let Err(e) = reload_hub_base_url_setting(conn, server_mode).await {
@@ -1901,6 +1903,24 @@ async fn process_notify_event(
                     .unwrap_or(false);
                     tracing::info!("Workspace telemetry setting changed: enabled={}", enabled);
                 }
+                RESTART_COORDINATION_SETTING => {
+                    // Internal coordination key for staggered restarts, no action needed
+                }
+                "plain_emails_telemetry" => {
+                    let enabled = sqlx::query_scalar!(
+                        "SELECT value FROM global_settings WHERE name = 'plain_emails_telemetry'"
+                    )
+                    .fetch_optional(db)
+                    .await
+                    .ok()
+                    .flatten()
+                    .and_then(|v| v.as_bool())
+                    .unwrap_or(false);
+                    tracing::info!(
+                        "Plain emails telemetry setting changed: enabled={}",
+                        enabled
+                    );
+                }
                 _ => {
                     tracing::info!("Unrecognized Global Setting Change Payload: {:?}", payload);
                 }
@@ -2042,14 +2062,145 @@ pub async fn run_workers(
     Ok(())
 }
 
-async fn send_delayed_killpill(tx: &KillpillSender, mut max_delay_secs: u64, context: &str) {
-    if max_delay_secs == 0 {
-        max_delay_secs = 1;
-    }
-    // Random delay to avoid all servers/workers shutting down simultaneously
-    let rd_delay = rand::rng().random_range(0..max_delay_secs);
-    tracing::info!("Scheduling {context} shutdown in {rd_delay}s");
-    tokio::time::sleep(Duration::from_secs(rd_delay)).await;
+/// Schedule a graceful restart with DB-coordinated staggering.
+///
+/// Uses a PostgreSQL advisory lock to serialize restart scheduling across server instances.
+/// Each instance records its planned restart time in the `_restart_coordination` global setting;
+/// subsequent instances read existing schedules and shift their restart to maintain at least
+/// `safety_margin_secs` between consecutive restarts (must exceed the server startup time).
+///
+/// Every server waits at least `DRAIN_DELAY_SECS` to let in-flight requests complete.
+/// Each subsequent server waits an additional `safety_margin_secs` after the previous one,
+/// guaranteeing zero downtime overlap.
+///
+/// The DB coordination is done synchronously (fast, ~ms) to reserve our restart slot,
+/// then the sleep+kill is spawned in the background so the notification handler is not blocked.
+///
+/// Falls back to drain-only delay if DB coordination fails.
+async fn spawn_graceful_killpill(
+    tx: &KillpillSender,
+    db: &Pool<Postgres>,
+    safety_margin_secs: u64,
+    context: &str,
+) {
+    // Minimum delay before any restart to let in-flight requests drain
+    const DRAIN_DELAY_SECS: u64 = 3;
 
-    tx.send();
+    let delay = match coordinate_restart_delay(db, safety_margin_secs, DRAIN_DELAY_SECS).await {
+        Ok(d) => d,
+        Err(e) => {
+            tracing::warn!(
+                "Failed to coordinate restart for {context}: {e:#}, \
+                 falling back to drain delay of {DRAIN_DELAY_SECS}s"
+            );
+            DRAIN_DELAY_SECS
+        }
+    };
+
+    tracing::info!("Scheduling {context} graceful shutdown in {delay}s");
+    let tx = tx.clone();
+    tokio::spawn(async move {
+        tokio::time::sleep(Duration::from_secs(delay)).await;
+        tx.send();
+    });
+}
+
+/// Coordinate a restart delay with other instances via the DB.
+///
+/// Returns the delay (in seconds from now) at which this instance should restart.
+/// The first server gets `drain_delay_secs` (to let in-flight requests complete).
+/// Each subsequent server is spaced `safety_margin_secs` after the latest scheduled restart.
+async fn coordinate_restart_delay(
+    db: &Pool<Postgres>,
+    safety_margin_secs: u64,
+    drain_delay_secs: u64,
+) -> anyhow::Result<u64> {
+    const RESTART_LOCK_ID: i64 = 737_483_920;
+    // Stale threshold: ignore coordination entries older than this
+    const STALE_THRESHOLD_SECS: i64 = 120;
+
+    let now = chrono::Utc::now();
+
+    let mut tx = db.begin().await.context("begin restart coordination tx")?;
+
+    // Serialize access across all instances
+    sqlx::query("SELECT pg_advisory_xact_lock($1)")
+        .bind(RESTART_LOCK_ID)
+        .execute(&mut *tx)
+        .await
+        .context("acquire restart coordination lock")?;
+
+    // Read existing coordination record
+    let existing: Option<serde_json::Value> =
+        sqlx::query_scalar("SELECT value FROM global_settings WHERE name = $1")
+            .bind(RESTART_COORDINATION_SETTING)
+            .fetch_optional(&mut *tx)
+            .await
+            .context("read restart coordination")?;
+
+    // Parse existing scheduled restarts, filtering out stale entries
+    // Each entry is (instance_name, restart_at)
+    let mut scheduled: Vec<(String, chrono::DateTime<chrono::Utc>)> = Vec::new();
+    if let Some(val) = &existing {
+        if let Some(arr) = val.get("restarts").and_then(|v| v.as_array()) {
+            for entry in arr {
+                let instance = entry
+                    .get("instance")
+                    .and_then(|v| v.as_str())
+                    .unwrap_or("unknown")
+                    .to_string();
+                if let Some(ts_str) = entry.get("restart_at").and_then(|v| v.as_str()) {
+                    if let Ok(dt) = chrono::DateTime::parse_from_rfc3339(ts_str) {
+                        let dt = dt.with_timezone(&chrono::Utc);
+                        let stale_cutoff = now - chrono::Duration::seconds(STALE_THRESHOLD_SECS);
+                        if dt > stale_cutoff {
+                            scheduled.push((instance, dt));
+                        }
+                    }
+                }
+            }
+        }
+    }
+
+    // Find the latest scheduled restart
+    let latest = scheduled.iter().map(|(_, dt)| *dt).max();
+    let earliest_allowed = now + chrono::Duration::seconds(drain_delay_secs as i64);
+
+    // Our restart time: drain_delay from now, or safety_margin after the latest existing restart
+    let our_restart = match latest {
+        Some(last) => {
+            let after_last = last + chrono::Duration::seconds(safety_margin_secs as i64);
+            // Use whichever is later: drain delay or staggered position
+            earliest_allowed.max(after_last)
+        }
+        None => earliest_allowed,
+    };
+
+    // Record our restart time (deduplicate: remove any prior entry for this instance)
+    scheduled.retain(|(inst, _)| inst != &*INSTANCE_NAME);
+    scheduled.push((INSTANCE_NAME.clone(), our_restart));
+    let new_value = serde_json::json!({
+        "restarts": scheduled.iter().map(|(inst, dt)| {
+            serde_json::json!({
+                "instance": inst,
+                "restart_at": dt.to_rfc3339()
+            })
+        }).collect::<Vec<_>>()
+    });
+
+    sqlx::query(
+        "INSERT INTO global_settings (name, value, updated_at) \
+         VALUES ($1, $2, now()) \
+         ON CONFLICT (name) DO UPDATE SET value = $2, updated_at = now()",
+    )
+    .bind(RESTART_COORDINATION_SETTING)
+    .bind(&new_value)
+    .execute(&mut *tx)
+    .await
+    .context("write restart coordination")?;
+
+    tx.commit().await.context("commit restart coordination")?;
+
+    let delay = (our_restart - now).num_seconds().max(0) as u64;
+    Ok(delay)
 }

backend/src/monitor.rs

@@ -169,6 +169,8 @@ lazy_static::lazy_static! {
 
     static ref QUEUE_COUNT_TAGS: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
     static ref QUEUE_RUNNING_COUNT_TAGS: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
+    static ref OTEL_QUEUE_COUNT_TAGS: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
+    static ref OTEL_QUEUE_RUNNING_COUNT_TAGS: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
     static ref DISABLE_CONCURRENCY_LIMIT: bool = std::env::var("DISABLE_CONCURRENCY_LIMIT").is_ok_and(|s| s == "true");
 
     //legacy typo
@@ -2372,8 +2374,20 @@ pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
             }
         }
 
+        let otel_enabled = OTEL_METRICS_ENABLED.load(Ordering::Relaxed);
+
+        if otel_enabled {
+            for q in OTEL_QUEUE_COUNT_TAGS.read().await.iter() {
+                if queue_counts.get(q).is_none() {
+                    otel_set_queue_count(q, 0);
+                }
+            }
+        }
+
         #[allow(unused_mut)]
         let mut tags_to_watch = vec![];
+        #[allow(unused_mut)]
+        let mut otel_tags_to_watch = vec![];
         for q in queue_counts {
             let count = q.1;
             let tag = q.0;
@@ -2385,6 +2399,9 @@ pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
                 tags_to_watch.push(tag.to_string());
             }
 
+            if otel_enabled {
+                otel_tags_to_watch.push(tag.to_string());
+            }
             otel_set_queue_count(&tag, count as i64);
 
             // save queue_count and delay metrics per tag
@@ -2419,9 +2436,13 @@ pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
             let mut w = QUEUE_COUNT_TAGS.write().await;
             *w = tags_to_watch;
         }
+        if otel_enabled {
+            let mut w = OTEL_QUEUE_COUNT_TAGS.write().await;
+            *w = otel_tags_to_watch;
+        }
 
         // Single DB query for running counts, shared by Prometheus and OTel
-        let otel_running = OTEL_METRICS_ENABLED.load(Ordering::Relaxed);
+        let otel_running = otel_enabled;
         #[cfg(feature = "prometheus")]
         let need_running_counts = metrics_enabled || otel_running;
         #[cfg(not(feature = "prometheus"))]
@@ -2439,8 +2460,18 @@ pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
                 }
             }
 
+            if otel_running {
+                for q in OTEL_QUEUE_RUNNING_COUNT_TAGS.read().await.iter() {
+                    if queue_running_counts.get(q).is_none() {
+                        otel_set_queue_running_count(q, 0);
+                    }
+                }
+            }
+
             #[allow(unused_mut, unused_variables)]
             let mut running_tags_to_watch: Vec<String> = vec![];
+            #[allow(unused_mut, unused_variables)]
+            let mut otel_running_tags_to_watch: Vec<String> = vec![];
             for (tag, count) in &queue_running_counts {
                 #[cfg(feature = "prometheus")]
                 if metrics_enabled {
@@ -2451,6 +2482,7 @@ pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
 
                 if otel_running {
                     otel_set_queue_running_count(tag, *count as i64);
+                    otel_running_tags_to_watch.push(tag.to_string());
                 }
             }
 
@@ -2459,6 +2491,10 @@ pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
                 let mut w = QUEUE_RUNNING_COUNT_TAGS.write().await;
                 *w = running_tags_to_watch;
             }
+            if otel_running {
+                let mut w = OTEL_QUEUE_RUNNING_COUNT_TAGS.write().await;
+                *w = otel_running_tags_to_watch;
+            }
         }
     }
 

backend/tests/otel.rs

@@ -0,0 +1,504 @@
+//! E2E tests for OpenTelemetry integration.
+//!
+//! Verify that metrics are recorded with correct names/values/attributes and
+//! spans are created with correct trace IDs, attributes, and status codes.
+//!
+//! Run with: cargo test --features enterprise,private,otel --test otel -- --test-threads=1
+
+#![cfg(all(feature = "otel", feature = "enterprise"))]
+
+use std::sync::{atomic::Ordering, Arc};
+
+use opentelemetry::global;
+use opentelemetry::trace::TracerProvider as _;
+use opentelemetry_sdk::{
+    metrics::{InMemoryMetricExporter, PeriodicReader, SdkMeterProvider},
+    trace::{InMemorySpanExporter, SdkTracerProvider, SimpleSpanProcessor},
+};
+use windmill_common::otel_ee::*;
+use windmill_common::{OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED};
+
+// ── Global test infrastructure ──────────────────────────────────────────
+
+struct OtelTestState {
+    metric_exporter: InMemoryMetricExporter,
+    span_exporter: InMemorySpanExporter,
+    meter_provider: SdkMeterProvider,
+}
+
+static STATE: tokio::sync::OnceCell<Arc<OtelTestState>> = tokio::sync::OnceCell::const_new();
+
+async fn ensure_setup() -> Arc<OtelTestState> {
+    STATE
+        .get_or_init(|| async {
+            // Metrics: InMemoryMetricExporter + PeriodicReader (needs async tokio context)
+            let metric_exporter = InMemoryMetricExporter::default();
+            let reader = PeriodicReader::builder(metric_exporter.clone()).build();
+            let meter_provider = SdkMeterProvider::builder().with_reader(reader).build();
+            global::set_meter_provider(meter_provider.clone());
+            OTEL_METRICS_ENABLED.store(true, Ordering::SeqCst);
+
+            // Tracing: InMemorySpanExporter + SimpleSpanProcessor
+            let span_exporter = InMemorySpanExporter::default();
+            let tracer_provider = SdkTracerProvider::builder()
+                .with_span_processor(SimpleSpanProcessor::new(span_exporter.clone()))
+                .build();
+            let tracer = tracer_provider.tracer("windmill");
+            *TRACER.write().unwrap() = Some(tracer);
+            OTEL_TRACING_ENABLED.store(true, Ordering::SeqCst);
+
+            Arc::new(OtelTestState { metric_exporter, span_exporter, meter_provider })
+        })
+        .await
+        .clone()
+}
+
+// ── Metric helper: flush + collect ──────────────────────────────────────
+
+fn flush_and_get_metrics(
+    state: &OtelTestState,
+) -> Vec<opentelemetry_sdk::metrics::data::ResourceMetrics> {
+    state.meter_provider.force_flush().expect("flush failed");
+    state
+        .metric_exporter
+        .get_finished_metrics()
+        .expect("get_finished_metrics failed")
+}
+
+fn find_metric<'a>(
+    all: &'a [opentelemetry_sdk::metrics::data::ResourceMetrics],
+    name: &str,
+) -> Option<&'a opentelemetry_sdk::metrics::data::Metric> {
+    all.iter()
+        .flat_map(|rm| rm.scope_metrics())
+        .flat_map(|sm| sm.metrics())
+        .find(|m| m.name() == name)
+}
+
+fn metric_names(all: &[opentelemetry_sdk::metrics::data::ResourceMetrics]) -> Vec<String> {
+    all.iter()
+        .flat_map(|rm| rm.scope_metrics())
+        .flat_map(|sm| sm.metrics())
+        .map(|m| m.name().to_string())
+        .collect()
+}
+
+// ── Counter value helpers ───────────────────────────────────────────────
+
+fn sum_u64_value(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option<u64> {
+    use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData};
+    match metric.data() {
+        AggregatedMetrics::U64(MetricData::Sum(sum)) => {
+            Some(sum.data_points().map(|dp| dp.value()).sum())
+        }
+        _ => None,
+    }
+}
+
+fn gauge_i64_values(
+    metric: &opentelemetry_sdk::metrics::data::Metric,
+) -> Vec<(Vec<opentelemetry::KeyValue>, i64)> {
+    use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData};
+    match metric.data() {
+        AggregatedMetrics::I64(MetricData::Gauge(gauge)) => gauge
+            .data_points()
+            .map(|dp| (dp.attributes().cloned().collect(), dp.value()))
+            .collect(),
+        _ => panic!("expected I64 Gauge metric"),
+    }
+}
+
+fn gauge_f64_value(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option<f64> {
+    use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData};
+    match metric.data() {
+        AggregatedMetrics::F64(MetricData::Gauge(gauge)) => {
+            gauge.data_points().next().map(|dp| dp.value())
+        }
+        _ => None,
+    }
+}
+
+fn histogram_f64_count(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option<u64> {
+    use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData};
+    match metric.data() {
+        AggregatedMetrics::F64(MetricData::Histogram(hist)) => {
+            Some(hist.data_points().map(|dp| dp.count()).sum())
+        }
+        _ => None,
+    }
+}
+
+fn histogram_f64_sum(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option<f64> {
+    use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData};
+    match metric.data() {
+        AggregatedMetrics::F64(MetricData::Histogram(hist)) => {
+            Some(hist.data_points().map(|dp| dp.sum()).sum())
+        }
+        _ => None,
+    }
+}
+
+// ═══════════════════════════════════════════════════════════════════════
+// METRICS E2E TEST
+//
+// All metric assertions live in one test function because the PeriodicReader's
+// background task is tied to the tokio runtime that created it. Separate
+// #[tokio::test] functions each get their own runtime, and the reader becomes
+// disconnected after the first test's runtime is dropped.
+// ═══════════════════════════════════════════════════════════════════════
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_all_metrics_e2e() {
+    let state = ensure_setup().await;
+
+    // ── Counters ────────────────────────────────────────────────────
+
+    otel_incr_queue_push_count();
+    otel_incr_queue_push_count();
+    otel_incr_queue_push_count();
+    otel_incr_queue_delete_count();
+    otel_incr_queue_pull_count();
+    otel_incr_zombie_restart_count(7);
+    otel_incr_zombie_delete_count(3);
+    otel_incr_worker_execution_count("bun");
+    otel_incr_worker_execution_count("bun");
+    otel_incr_worker_execution_failed("go");
+    otel_incr_worker_started();
+
+    // ── Gauges ──────────────────────────────────────────────────────
+
+    otel_set_queue_count("python3", 42);
+    otel_set_queue_running_count("deno", 5);
+    otel_set_worker_busy("worker-test-1", 1);
+    otel_set_db_pool(5, 10, 20);
+    otel_set_health_db_latency(2.5);
+    otel_set_worker_uptime("w-uptime", 3600.0);
+    otel_set_health_status_phase("healthy");
+    otel_set_health_db_unresponsive(true);
+
+    // ── Histograms ──────────────────────────────────────────────────
+
+    otel_record_worker_execution_duration("python3", 1.5);
+    otel_record_worker_execution_duration("python3", 2.5);
+    otel_record_worker_pull_duration("w1", true, 0.05);
+    otel_record_worker_pull_duration("w1", false, 0.01);
+
+    // ── Flush and collect ───────────────────────────────────────────
+
+    let metrics = flush_and_get_metrics(&state);
+    let names = metric_names(&metrics);
+
+    // ── Verify all 20 metric names are present ──────────────────────
+
+    let expected = [
+        "windmill.queue.push_count",
+        "windmill.queue.delete_count",
+        "windmill.queue.pull_count",
+        "windmill.queue.zombie_restart_count",
+        "windmill.queue.zombie_delete_count",
+        "windmill.queue.count",
+        "windmill.queue.running_count",
+        "windmill.worker.execution_count",
+        "windmill.worker.execution_duration",
+        "windmill.worker.busy",
+        "windmill.worker.pull_duration",
+        "windmill.worker.execution_failed",
+        "windmill.db.pool.active",
+        "windmill.db.pool.idle",
+        "windmill.db.pool.max",
+        "windmill.health.db_latency",
+        "windmill.worker.started",
+        "windmill.worker.uptime",
+        "windmill.health.status",
+        "windmill.health.db_unresponsive",
+    ];
+    for name in expected {
+        assert!(
+            names.iter().any(|n| n == name),
+            "metric '{}' not found in {:?}",
+            name,
+            names
+        );
+    }
+
+    // ── Counter values ──────────────────────────────────────────────
+
+    let m = find_metric(&metrics, "windmill.queue.push_count").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 3, "push_count should be >= 3");
+
+    let m = find_metric(&metrics, "windmill.queue.delete_count").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 1);
+
+    let m = find_metric(&metrics, "windmill.queue.pull_count").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 1);
+
+    let m = find_metric(&metrics, "windmill.queue.zombie_restart_count").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 7);
+
+    let m = find_metric(&metrics, "windmill.queue.zombie_delete_count").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 3);
+
+    let m = find_metric(&metrics, "windmill.worker.execution_count").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 2);
+
+    let m = find_metric(&metrics, "windmill.worker.execution_failed").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 1);
+
+    let m = find_metric(&metrics, "windmill.worker.started").unwrap();
+    assert!(sum_u64_value(m).unwrap() >= 1);
+
+    // ── Gauge values ────────────────────────────────────────────────
+
+    let m = find_metric(&metrics, "windmill.queue.count").unwrap();
+    let values = gauge_i64_values(m);
+    let dp = values
+        .iter()
+        .find(|(attrs, _)| {
+            attrs
+                .iter()
+                .any(|kv| kv.key.as_str() == "tag" && kv.value.as_str() == "python3")
+        })
+        .expect("queue.count data point with tag=python3 not found");
+    assert_eq!(dp.1, 42);
+
+    let m = find_metric(&metrics, "windmill.queue.running_count").unwrap();
+    let values = gauge_i64_values(m);
+    let dp = values
+        .iter()
+        .find(|(attrs, _)| {
+            attrs
+                .iter()
+                .any(|kv| kv.key.as_str() == "tag" && kv.value.as_str() == "deno")
+        })
+        .expect("running_count data point with tag=deno not found");
+    assert_eq!(dp.1, 5);
+
+    let m = find_metric(&metrics, "windmill.worker.busy").unwrap();
+    let values = gauge_i64_values(m);
+    let dp = values
+        .iter()
+        .find(|(attrs, _)| {
+            attrs
+                .iter()
+                .any(|kv| kv.key.as_str() == "worker" && kv.value.as_str() == "worker-test-1")
+        })
+        .expect("worker.busy data point with worker=worker-test-1 not found");
+    assert_eq!(dp.1, 1);
+
+    let m = find_metric(&metrics, "windmill.db.pool.active").unwrap();
+    assert_eq!(gauge_i64_values(m)[0].1, 5);
+    let m = find_metric(&metrics, "windmill.db.pool.idle").unwrap();
+    assert_eq!(gauge_i64_values(m)[0].1, 10);
+    let m = find_metric(&metrics, "windmill.db.pool.max").unwrap();
+    assert_eq!(gauge_i64_values(m)[0].1, 20);
+
+    let m = find_metric(&metrics, "windmill.health.db_latency").unwrap();
+    assert!((gauge_f64_value(m).unwrap() - 2.5).abs() < f64::EPSILON);
+
+    let m = find_metric(&metrics, "windmill.worker.uptime").unwrap();
+    assert!((gauge_f64_value(m).unwrap() - 3600.0).abs() < f64::EPSILON);
+
+    let m = find_metric(&metrics, "windmill.health.db_unresponsive").unwrap();
+    assert_eq!(gauge_i64_values(m)[0].1, 1);
+
+    // ── Health status phase (all 3 phases) ──────────────────────────
+
+    let m = find_metric(&metrics, "windmill.health.status").unwrap();
+    let values = gauge_i64_values(m);
+    let healthy = values
+        .iter()
+        .find(|(attrs, _)| {
+            attrs
+                .iter()
+                .any(|kv| kv.key.as_str() == "phase" && kv.value.as_str() == "healthy")
+        })
+        .expect("phase=healthy");
+    let degraded = values
+        .iter()
+        .find(|(attrs, _)| {
+            attrs
+                .iter()
+                .any(|kv| kv.key.as_str() == "phase" && kv.value.as_str() == "degraded")
+        })
+        .expect("phase=degraded");
+    let unhealthy = values
+        .iter()
+        .find(|(attrs, _)| {
+            attrs
+                .iter()
+                .any(|kv| kv.key.as_str() == "phase" && kv.value.as_str() == "unhealthy")
+        })
+        .expect("phase=unhealthy");
+    assert_eq!(healthy.1, 1);
+    assert_eq!(degraded.1, 0);
+    assert_eq!(unhealthy.1, 0);
+
+    // ── Histogram values ────────────────────────────────────────────
+
+    let m = find_metric(&metrics, "windmill.worker.execution_duration").unwrap();
+    assert!(histogram_f64_count(m).unwrap() >= 2);
+    assert!(histogram_f64_sum(m).unwrap() >= 4.0);
+
+    let m = find_metric(&metrics, "windmill.worker.pull_duration").unwrap();
+    assert!(histogram_f64_count(m).unwrap() >= 2);
+}
+
+// ═══════════════════════════════════════════════════════════════════════
+// SPAN E2E TESTS
+// ═══════════════════════════════════════════════════════════════════════
+
+fn make_test_job(id: uuid::Uuid, parent: Option<uuid::Uuid>) -> windmill_queue::MiniPulledJob {
+    use windmill_types::jobs::JobKind;
+    let mut job = windmill_queue::MiniPulledJob::new_inline(
+        "test-workspace".to_string(),
+        None,
+        "test-user".to_string(),
+        "u/test-user".to_string(),
+        "test@example.com".to_string(),
+        Some("f/test/script".to_string()),
+        JobKind::Script,
+        None,
+        "deno".to_string(),
+        None,
+    );
+    job.id = id;
+    job.parent_job = parent;
+    job.started_at = Some(chrono::Utc::now());
+    job
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_root_job_span_created_on_success() {
+    let state = ensure_setup().await;
+    state.span_exporter.reset();
+
+    let job_id = uuid::Uuid::new_v4();
+    let job = make_test_job(job_id, None);
+    windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true);
+
+    let spans = state.span_exporter.get_finished_spans().unwrap();
+    let span = spans
+        .iter()
+        .find(|s| s.name == "full_job")
+        .expect("full_job span not found");
+
+    assert_eq!(span.status, opentelemetry::trace::Status::Ok,);
+
+    // Verify attributes
+    let attrs: Vec<_> = span.attributes.iter().map(|kv| kv.key.as_str()).collect();
+    assert!(attrs.contains(&"job_id"), "missing job_id attribute");
+    assert!(
+        attrs.contains(&"workspace_id"),
+        "missing workspace_id attribute"
+    );
+    assert!(
+        attrs.contains(&"script_path"),
+        "missing script_path attribute"
+    );
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_root_job_span_error_on_failure() {
+    let state = ensure_setup().await;
+    state.span_exporter.reset();
+
+    let job_id = uuid::Uuid::new_v4();
+    let job = make_test_job(job_id, None);
+    windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, false);
+
+    let spans = state.span_exporter.get_finished_spans().unwrap();
+    let span = spans
+        .iter()
+        .find(|s| s.name == "full_job")
+        .expect("full_job span not found");
+
+    match &span.status {
+        opentelemetry::trace::Status::Error { description } => {
+            assert_eq!(description.as_ref(), "Job failed");
+        }
+        other => panic!("expected Error status, got {:?}", other),
+    }
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_root_job_trace_id_matches_uuid() {
+    let state = ensure_setup().await;
+    state.span_exporter.reset();
+
+    let job_id = uuid::Uuid::parse_str("550e8400-e29b-41d4-a716-446655440000").unwrap();
+    let job = make_test_job(job_id, None);
+    windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true);
+
+    let spans = state.span_exporter.get_finished_spans().unwrap();
+    let span = spans
+        .iter()
+        .find(|s| s.name == "full_job")
+        .expect("full_job span not found");
+
+    let expected_trace_id =
+        opentelemetry::trace::TraceId::from_bytes(job_id.as_u128().to_be_bytes());
+    assert_eq!(span.span_context.trace_id(), expected_trace_id);
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_root_job_span_id_matches_uuid() {
+    let state = ensure_setup().await;
+    state.span_exporter.reset();
+
+    let job_id = uuid::Uuid::parse_str("550e8400-e29b-41d4-a716-446655440000").unwrap();
+    let job = make_test_job(job_id, None);
+    windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true);
+
+    let spans = state.span_exporter.get_finished_spans().unwrap();
+    let span = spans
+        .iter()
+        .find(|s| s.name == "full_job")
+        .expect("full_job span not found");
+
+    let expected_span_id =
+        opentelemetry::trace::SpanId::from_bytes(job_id.as_u64_pair().1.to_be_bytes());
+    assert_eq!(span.span_context.span_id(), expected_span_id);
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_child_job_produces_no_span() {
+    let state = ensure_setup().await;
+    state.span_exporter.reset();
+
+    let parent_id = uuid::Uuid::new_v4();
+    let job_id = uuid::Uuid::new_v4();
+    let job = make_test_job(job_id, Some(parent_id));
+    windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true);
+
+    let spans = state.span_exporter.get_finished_spans().unwrap();
+    let found = spans.iter().any(|s| s.name == "full_job");
+    assert!(!found, "child job should not produce a span");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_root_job_span_attributes_values() {
+    let state = ensure_setup().await;
+    state.span_exporter.reset();
+
+    let job_id = uuid::Uuid::new_v4();
+    let job = make_test_job(job_id, None);
+    windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true);
+
+    let spans = state.span_exporter.get_finished_spans().unwrap();
+    let span = spans
+        .iter()
+        .find(|s| s.name == "full_job")
+        .expect("full_job span not found");
+
+    let get_attr = |key: &str| -> String {
+        span.attributes
+            .iter()
+            .find(|kv| kv.key.as_str() == key)
+            .map(|kv| kv.value.as_str().to_string())
+            .unwrap_or_default()
+    };
+
+    assert_eq!(get_attr("job_id"), job_id.to_string());
+    assert_eq!(get_attr("workspace_id"), "test-workspace");
+    assert_eq!(get_attr("script_path"), "f/test/script");
+}

backend/windmill-api-auth/src/auth.rs

@@ -1,7 +1,6 @@
 #[cfg(feature = "enterprise")]
 use crate::ee_oss::ExternalJwks;
 use axum::{
-    async_trait,
     extract::{FromRequestParts, OriginalUri, Query},
     Extension, Json,
 };
@@ -226,7 +225,15 @@ impl AuthCache {
                     t_hash,
                     w_id.as_ref(),
                 )
-                .map(|x| (x.owner, x.email, x.super_admin, x.scopes, x.label))
+                .map(|x| {
+                    (
+                        x.owner,
+                        x.email,
+                        x.super_admin,
+                        x.scopes,
+                        x.label,
+                    )
+                })
                 .fetch_optional(&self.db)
                 .await
                 .ok()
@@ -235,7 +242,13 @@ impl AuthCache {
                 if let Some(user) = user_o {
                     let authed_o = {
                         match user {
-                            (Some(owner), Some(email), super_admin, _, label) if w_id.is_some() => {
+                            (
+                                Some(owner),
+                                Some(email),
+                                super_admin,
+                                _,
+                                label,
+                            ) if w_id.is_some() => {
                                 let username_override = username_override_from_label(label);
                                 if let Some((prefix, name)) = owner.split_once('/') {
                                     if prefix == "u" {
@@ -451,7 +464,11 @@ pub(crate) async fn extract_token<S: Send + Sync>(parts: &mut Parts, state: &S)
         None => Extension::<Cookies>::from_request_parts(parts, state)
             .await
             .ok()
-            .and_then(|cookies| cookies.get(COOKIE_NAME).map(|c| c.value().to_owned())),
+            .and_then(|cookies| {
+                cookies
+                    .get(COOKIE_NAME)
+                    .map(|c| c.value_trimmed().to_owned())
+            }),
     };
 
     #[derive(Deserialize)]
@@ -504,7 +521,6 @@ impl BruteForceCounter {
     }
 }
 
-#[async_trait]
 impl<S> FromRequestParts<S> for Tokened
 where
     S: Send + Sync,
@@ -535,7 +551,6 @@ where
     }
 }
 
-#[async_trait]
 impl<S> FromRequestParts<S> for OptTokened
 where
     S: Send + Sync,

backend/windmill-api-auth/src/lib.rs

@@ -12,8 +12,7 @@ pub mod ee;
 pub mod ee_oss;
 pub mod scopes;
 
-use axum::async_trait;
-use axum::extract::FromRequestParts;
+use axum::extract::{FromRequestParts, OptionalFromRequestParts};
 use http::request::Parts;
 
 use windmill_audit::audit_oss::AuditAuthorable;
@@ -345,7 +344,6 @@ pub async fn maybe_refresh_folders(
 
 // ------------ FromRequestParts impls (direct call to auth module) ------------
 
-#[async_trait]
 impl<S> FromRequestParts<S> for ApiAuthed
 where
     S: Send + Sync,
@@ -361,7 +359,24 @@ where
     }
 }
 
-#[async_trait]
+impl<S> OptionalFromRequestParts<S> for ApiAuthed
+where
+    S: Send + Sync,
+{
+    type Rejection = std::convert::Infallible;
+
+    async fn from_request_parts(
+        parts: &mut Parts,
+        state: &S,
+    ) -> std::result::Result<Option<Self>, Self::Rejection> {
+        Ok(
+            <Self as FromRequestParts<S>>::from_request_parts(parts, state)
+                .await
+                .ok(),
+        )
+    }
+}
+
 impl<S> FromRequestParts<S> for OptJobAuthed
 where
     S: Send + Sync,
@@ -397,7 +412,6 @@ fn empty_parts() -> Parts {
 #[derive(Clone, Debug)]
 pub struct OptAuthed(pub Option<ApiAuthed>);
 
-#[async_trait]
 impl<S> FromRequestParts<S> for OptAuthed
 where
     S: Send + Sync,
@@ -408,7 +422,7 @@ where
         parts: &mut Parts,
         state: &S,
     ) -> std::result::Result<Self, Self::Rejection> {
-        ApiAuthed::from_request_parts(parts, state)
+        <ApiAuthed as FromRequestParts<S>>::from_request_parts(parts, state)
             .await
             .map(|authed| Self(Some(authed)))
             .or_else(|_| Ok(Self(None)))

backend/windmill-api-auth/src/scopes.rs

@@ -442,9 +442,22 @@ pub fn check_route_access(
     // Find the domain and kind for this route
     let (required_domain, required_kind, route_suffix) = extract_domain_from_route(route_path)?;
 
-    // Backward compatibility: MCP handlers expect unusual scope actions: all, favorites, hub.
+    // MCP scopes (mcp:all, mcp:favorites, mcp:hub:*, etc.) use a custom format
+    // that doesn't fit the standard domain:action model. Verify the token has at
+    // least one mcp: scope; MCP handlers do their own fine-grained checking.
     if required_domain == ScopeDomain::Mcp {
-        return Ok(());
+        let is_scoped_token = token_scopes
+            .iter()
+            .any(|s| !s.starts_with("if_jobs:filter_tags:"));
+        if !is_scoped_token {
+            return Ok(());
+        }
+        if token_scopes.iter().any(|s| s.starts_with("mcp:")) {
+            return Ok(());
+        }
+        return Err(Error::NotAuthorized(
+            "Access denied. Required scope: mcp:*".to_string(),
+        ));
     }
 
     // tracing::error!("Checking route access {:?} {:?} {:?} {:?}", required_action, required_domain, required_kind, route_suffix);
@@ -931,4 +944,50 @@ mod tests {
             ScopeDefinition::new("scripts", "read", None, Some(vec!["u/*".to_string()]));
         assert!(scope_specific_path.includes(&required_broad));
     }
+
+    #[test]
+    fn test_mcp_scope_bypass_blocked_without_mcp_scope() {
+        // A token with only jobs:read should NOT be able to access MCP endpoints
+        let scopes = vec!["jobs:read".to_string()];
+        assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_err());
+        assert!(
+            check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "POST").is_err()
+        );
+    }
+
+    #[test]
+    fn test_mcp_scope_allowed_with_mcp_scope() {
+        // A token with mcp:all should access MCP endpoints
+        let scopes = vec!["mcp:all".to_string()];
+        assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok());
+
+        // mcp:favorites should also work
+        let scopes = vec!["mcp:favorites".to_string()];
+        assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "POST").is_ok());
+
+        // mcp:scripts:path should also work
+        let scopes = vec!["mcp:scripts:u/admin/script1".to_string()];
+        assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok());
+    }
+
+    #[test]
+    fn test_mcp_scope_filter_tags_only_treated_as_unrestricted() {
+        // Token with only filter_tags is not considered scoped — should be allowed
+        let scopes = vec!["if_jobs:filter_tags:tag1".to_string()];
+        assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok());
+    }
+
+    #[test]
+    fn test_mcp_scope_mixed_scopes_without_mcp() {
+        // Token with multiple non-MCP scopes should be denied
+        let scopes = vec!["jobs:read".to_string(), "scripts:write".to_string()];
+        assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_err());
+    }
+
+    #[test]
+    fn test_mcp_scope_mixed_scopes_with_mcp() {
+        // Token with MCP scope + other scopes should be allowed for MCP
+        let scopes = vec!["jobs:read".to_string(), "mcp:all".to_string()];
+        assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok());
+    }
 }

backend/windmill-api-configs/src/lib.rs

@@ -28,11 +28,11 @@ use windmill_api_auth::{require_devops_role, ApiAuthed};
 pub fn global_service() -> Router {
     Router::new()
         .route("/list_worker_groups", get(list_worker_groups))
-        .route("/update/:name", post(update_config).delete(delete_config))
-        .route("/get/:name", get(get_config))
+        .route("/update/{name}", post(update_config).delete(delete_config))
+        .route("/get/{name}", get(get_config))
         .route("/list", get(list_configs))
         .route(
-            "/list_autoscaling_events/:worker_group",
+            "/list_autoscaling_events/{worker_group}",
             get(list_autoscaling_events),
         )
         .route(

backend/windmill-api-flow-conversations/src/lib.rs

@@ -21,8 +21,8 @@ use windmill_common::{
 pub fn workspaced_service() -> Router {
     Router::new()
         .route("/list", get(list_conversations))
-        .route("/delete/:conversation_id", delete(delete_conversation))
-        .route("/:conversation_id/messages", get(list_messages))
+        .route("/delete/{conversation_id}", delete(delete_conversation))
+        .route("/{conversation_id}/messages", get(list_messages))
 }
 
 #[derive(Serialize, FromRow, Debug)]

backend/windmill-api-flows/src/flows.rs

@@ -61,26 +61,26 @@ pub fn workspaced_service() -> Router {
         .route("/list", get(list_flows))
         .route("/list_search", get(list_search_flows))
         .route("/create", post(create_flow))
-        .route("/update/*path", post(update_flow))
-        .route("/archive/*path", post(archive_flow_by_path))
-        .route("/delete/*path", delete(delete_flow_by_path))
-        .route("/list_tokens/*path", get(list_tokens))
-        .route("/get/*path", get(get_flow_by_path))
-        .route("/deployment_status/p/*path", get(get_deployment_status))
-        .route("/get/draft/*path", get(get_flow_by_path_w_draft))
-        .route("/exists/*path", get(exists_flow_by_path))
+        .route("/update/{*path}", post(update_flow))
+        .route("/archive/{*path}", post(archive_flow_by_path))
+        .route("/delete/{*path}", delete(delete_flow_by_path))
+        .route("/list_tokens/{*path}", get(list_tokens))
+        .route("/get/{*path}", get(get_flow_by_path))
+        .route("/deployment_status/p/{*path}", get(get_deployment_status))
+        .route("/get/draft/{*path}", get(get_flow_by_path_w_draft))
+        .route("/exists/{*path}", get(exists_flow_by_path))
         .route("/list_paths", get(list_paths))
-        .route("/history/p/*path", get(get_flow_history))
-        .route("/get_latest_version/*path", get(get_latest_version))
+        .route("/history/p/{*path}", get(get_flow_history))
+        .route("/get_latest_version/{*path}", get(get_latest_version))
         .route(
-            "/list_paths_from_workspace_runnable/:runnable_kind/*path",
+            "/list_paths_from_workspace_runnable/{runnable_kind}/{*path}",
             get(list_paths_from_workspace_runnable),
         )
-        .route("/history_update/v/:version", post(update_flow_history))
-        .route("/get/v/:version", get(get_flow_version_by_id))
-        .route("/get/v/:version/p/*path", get(get_flow_version))
+        .route("/history_update/v/{version}", post(update_flow_history))
+        .route("/get/v/{version}", get(get_flow_version_by_id))
+        .route("/get/v/{version}/p/{*path}", get(get_flow_version))
         .route(
-            "/toggle_workspace_error_handler/*path",
+            "/toggle_workspace_error_handler/{*path}",
             post(toggle_workspace_error_handler),
         )
 }
@@ -88,7 +88,7 @@ pub fn workspaced_service() -> Router {
 pub fn global_service() -> Router {
     Router::new()
         .route("/hub/list", get(list_hub_flows))
-        .route("/hub/get/:id", get(get_hub_flow_by_id))
+        .route("/hub/get/{id}", get(get_hub_flow_by_id))
 }
 
 #[derive(Serialize, FromRow)]

backend/windmill-api-groups/src/folder_history.rs

@@ -22,7 +22,7 @@ use serde::Serialize;
 use sqlx::FromRow;
 
 pub fn workspaced_service() -> Router {
-    Router::new().route("/get/:name", get(get_folder_permission_history))
+    Router::new().route("/get/{name}", get(get_folder_permission_history))
 }
 
 #[derive(Serialize, FromRow)]

backend/windmill-api-groups/src/folders.rs

@@ -40,14 +40,14 @@ pub fn workspaced_service() -> Router {
         .route("/list", get(list_folders))
         .route("/listnames", get(list_foldernames))
         .route("/create", post(create_folder))
-        .route("/get/:name", get(get_folder))
-        .route("/exists/:name", get(exists_folder))
-        .route("/update/:name", post(update_folder))
-        .route("/getusage/:name", get(get_folder_usage))
-        .route("/delete/:name", delete(delete_folder))
-        .route("/addowner/:name", post(add_owner))
-        .route("/removeowner/:name", post(remove_owner))
-        .route("/is_owner/*path", get(is_owner_api))
+        .route("/get/{name}", get(get_folder))
+        .route("/exists/{name}", get(exists_folder))
+        .route("/update/{name}", post(update_folder))
+        .route("/getusage/{name}", get(get_folder_usage))
+        .route("/delete/{name}", delete(delete_folder))
+        .route("/addowner/{name}", post(add_owner))
+        .route("/removeowner/{name}", post(remove_owner))
+        .route("/is_owner/{*path}", get(is_owner_api))
 }
 
 #[derive(FromRow, Serialize, Deserialize, Clone)]

backend/windmill-api-groups/src/granular_acls.rs

@@ -48,9 +48,9 @@ const KINDS: [&str; 19] = [
 
 pub fn workspaced_service() -> Router {
     Router::new()
-        .route("/get/*path", get(get_granular_acls))
-        .route("/add/*path", post(add_granular_acl))
-        .route("/remove/*path", post(remove_granular_acl))
+        .route("/get/{*path}", get(get_granular_acls))
+        .route("/add/{*path}", post(add_granular_acl))
+        .route("/remove/{*path}", post(remove_granular_acl))
 }
 
 #[derive(Serialize, Deserialize)]

backend/windmill-api-groups/src/groups.rs

@@ -33,24 +33,24 @@ pub fn workspaced_service() -> Router {
         .route("/list", get(list_groups))
         .route("/listnames", get(list_group_names))
         .route("/create", post(create_group))
-        .route("/get/:name", get(get_group))
-        .route("/update/:name", post(update_group))
-        .route("/delete/:name", delete(delete_group))
-        .route("/adduser/:name", post(add_user))
-        .route("/removeuser/:name", post(remove_user))
-        .route("/is_owner/:name", get(is_owner))
+        .route("/get/{name}", get(get_group))
+        .route("/update/{name}", post(update_group))
+        .route("/delete/{name}", delete(delete_group))
+        .route("/adduser/{name}", post(add_user))
+        .route("/removeuser/{name}", post(remove_user))
+        .route("/is_owner/{name}", get(is_owner))
 }
 
 pub fn global_service() -> Router {
     Router::new()
         .route("/list", get(list_igroups))
         .route("/list_with_workspaces", get(list_igroups_with_workspaces))
-        .route("/get/:name", get(get_igroup))
+        .route("/get/{name}", get(get_igroup))
         .route("/create", post(create_igroup))
-        .route("/update/:name", post(update_igroup))
-        .route("/delete/:name", delete(delete_igroup))
-        .route("/adduser/:name", post(add_user_igroup))
-        .route("/removeuser/:name", post(remove_user_igroup))
+        .route("/update/{name}", post(update_igroup))
+        .route("/delete/{name}", delete(delete_igroup))
+        .route("/adduser/{name}", post(add_user_igroup))
+        .route("/removeuser/{name}", post(remove_user_igroup))
         .route("/export", get(export_igroups))
         .route("/overwrite", post(overwrite_igroups))
 }
@@ -851,9 +851,21 @@ async fn add_user_igroup(
     #[cfg(all(feature = "private", feature = "enterprise"))]
     {
         use windmill_api_workspaces::workspaces_ee::auto_add_user;
+        use windmill_common::users::compute_highest_workspace_role;
+
+        // Find all instance groups this user belongs to (includes the newly added group)
+        let user_igroups: Vec<String> = sqlx::query_scalar!(
+            "SELECT igroup FROM email_to_igroup WHERE email = $1",
+            &email
+        )
+        .fetch_all(&mut *tx)
+        .await?;
+
         let workspaces = sqlx::query!(
             r#"
-            SELECT workspace_id, auto_invite->'instance_groups_roles' as instance_groups_roles
+            SELECT workspace_id,
+                   auto_invite->'instance_groups_roles' as instance_groups_roles,
+                   auto_invite->'instance_groups' as instance_groups_json
             FROM workspace_settings
             WHERE auto_invite->'instance_groups' ? $1
             "#,
@@ -861,34 +873,53 @@ async fn add_user_igroup(
         )
         .fetch_all(&mut *tx)
         .await?;
+
         for ws in workspaces {
-            let role = ws
+            let roles: std::collections::HashMap<String, String> = ws
                 .instance_groups_roles
-                .and_then(|r| r.get(&name).and_then(|v| v.as_str().map(String::from)))
-                .unwrap_or_else(|| "developer".to_string());
-            let (is_admin, is_operator) = match role.as_str() {
-                "admin" => (true, false),
-                "operator" => (false, true),
-                _ => (false, false),
-            };
+                .and_then(|r| serde_json::from_value(r).ok())
+                .unwrap_or_default();
+
+            let ws_configured_groups: Vec<String> = ws
+                .instance_groups_json
+                .and_then(|ig| serde_json::from_value(ig).ok())
+                .unwrap_or_default();
+
+            let (best_group, is_admin, is_operator) =
+                compute_highest_workspace_role(&user_igroups, &ws_configured_groups, &roles);
+
+            let instance_group_source = serde_json::json!({
+                "source": "instance_group",
+                "group": &best_group
+            });
+
+            // auto_add_user creates the user if they don't exist (ON CONFLICT DO NOTHING).
+            // The operator flag here doesn't matter for the final state — the UPDATE below
+            // always sets the correct is_admin/operator based on the highest-precedence role.
             auto_add_user(
                 &email,
                 &ws.workspace_id,
-                &is_operator,
+                &false,
                 &mut tx,
                 &authed,
-                Some(serde_json::json!({"source": "instance_group", "group": &name})),
+                Some(instance_group_source.clone()),
             )
             .await?;
-            if is_admin {
-                sqlx::query!(
-                    "UPDATE usr SET is_admin = true WHERE workspace_id = $1 AND email = $2",
-                    &ws.workspace_id,
-                    &email
-                )
-                .execute(&mut *tx)
-                .await?;
-            }
+
+            // Set the correct role based on highest precedence across all groups.
+            // For new users, auto_add_user already stored added_via with source=instance_group,
+            // so this UPDATE will match. For existing instance_group users, it upgrades/corrects
+            // the role. Manually-added users (added_via is NULL or non-instance_group) are not affected.
+            sqlx::query!(
+                "UPDATE usr SET is_admin = $1, operator = $2, added_via = $3 WHERE workspace_id = $4 AND email = $5 AND added_via->>'source' = 'instance_group'",
+                is_admin,
+                is_operator,
+                &instance_group_source,
+                &ws.workspace_id,
+                &email
+            )
+            .execute(&mut *tx)
+            .await?;
         }
     }
 

backend/windmill-api-inputs/src/lib.rs

@@ -26,6 +26,7 @@ use windmill_common::{
     jobs::JobKind,
     scripts::to_i64,
     utils::{not_found_if_none, paginate, Pagination},
+    worker::CLOUD_HOSTED,
 };
 pub fn workspaced_service() -> Router {
     Router::new()
@@ -33,9 +34,9 @@ pub fn workspaced_service() -> Router {
         .route("/list", get(list_saved_inputs))
         .route("/create", post(create_input))
         .route("/update", post(update_input))
-        .route("/delete/:id", post(delete_input))
+        .route("/delete/{id}", post(delete_input))
         .route(
-            "/:job_or_input_id/args",
+            "/{job_or_input_id}/args",
             get(get_args_from_history_or_saved_input),
         )
 }
@@ -134,11 +135,20 @@ async fn get_input_history(
     Query(g): Query<GetInputHistory>,
 ) -> JsonResult<Vec<Input>> {
     let (per_page, offset) = paginate(pagination);
+    let per_page = if *CLOUD_HOSTED {
+        per_page.min(100)
+    } else {
+        per_page
+    };
 
     let mut tx = user_db.begin(&authed).await?;
 
     let args_query = if let Some(args) = &g.args {
-        sql_builder::bind::Bind::bind(&"and v2_job.args @> ?", &args.replace("'", "''"))
+        if let Ok(v) = serde_json::from_str::<serde_json::Value>(args) {
+            sql_builder::bind::Bind::bind(&"and v2_job.args @> ?", &v.to_string())
+        } else {
+            "AND FALSE".to_string()
+        }
     } else {
         "".to_string()
     };

backend/windmill-api-integration-tests/Cargo.toml

@@ -13,7 +13,7 @@ default = []
 private = ["windmill-test-utils/private", "dep:aws-config", "dep:aws-credential-types", "dep:aws-sdk-sqs", "windmill-git-sync/private"]
 enterprise = ["windmill-test-utils/enterprise", "dep:base64", "windmill-git-sync/enterprise"]
 deno_core = ["windmill-test-utils/deno_core"]
-mcp = []
+mcp = ["windmill-test-utils/mcp", "dep:rmcp"]
 run_inline = ["dep:windmill-worker", "windmill-test-utils/run_inline", "windmill-test-utils/duckdb"]
 
 [dependencies]
@@ -40,3 +40,5 @@ aws-config = { workspace = true, optional = true }
 aws-credential-types = { workspace = true, optional = true }
 aws-sdk-sqs = { workspace = true, optional = true }
 base64 = { workspace = true, optional = true }
+axum.workspace = true
+rmcp = { workspace = true, optional = true }

backend/windmill-api-integration-tests/tests/ai_routes.rs

@@ -0,0 +1,106 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+/// Start a mock AI API that echoes back a valid chat completion response.
+async fn start_mock_ai_api() -> u16 {
+    use axum::{routing::post, Json, Router};
+
+    let app = Router::new().fallback(post(|| async {
+        Json(json!({
+            "id": "chatcmpl-test",
+            "object": "chat.completion",
+            "choices": [{"message": {"role": "assistant", "content": "hello"}}]
+        }))
+    }));
+    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+    let port = listener.local_addr().unwrap().port();
+    tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
+    port
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_ai_proxy_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    // Start mock AI API
+    let mock_port = start_mock_ai_api().await;
+    let mock_url = format!("http://127.0.0.1:{mock_port}/v1");
+
+    // Create an openai resource pointing to the mock
+    let resp = authed(
+        client()
+            .post(format!(
+                "http://localhost:{port}/api/w/test-workspace/resources/create"
+            ))
+            .json(&json!({
+                "path": "f/ai/openai_config",
+                "resource_type": "openai",
+                "value": {
+                    "api_key": "test-key",
+                    "base_url": mock_url
+                }
+            })),
+    )
+    .send()
+    .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "create openai resource",
+    );
+
+    // Set ai_config on workspace_settings directly via SQL
+    sqlx::query(
+        "UPDATE workspace_settings SET ai_config = $1::jsonb WHERE workspace_id = 'test-workspace'",
+    )
+    .bind(json!({
+        "providers": {
+            "openai": {
+                "resource_path": "f/ai/openai_config",
+                "models": ["gpt-4"]
+            }
+        }
+    }))
+    .execute(&db)
+    .await?;
+
+    // POST /w/{ws}/ai/proxy/chat/completions
+    let resp = authed(
+        client()
+            .post(format!(
+                "http://localhost:{port}/api/w/test-workspace/ai/proxy/chat/completions"
+            ))
+            .header("X-Provider", "openai")
+            .json(&json!({
+                "model": "gpt-4",
+                "messages": [{"role": "user", "content": "hi"}]
+            })),
+    )
+    .send()
+    .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /ai/proxy/chat/completions",
+    );
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/audit.rs

@@ -0,0 +1,35 @@
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_audit_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/audit");
+
+    // GET /list returns 200 (empty array)
+    let resp = authed(client().get(format!("{base}/list"))).send().await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /audit/list",
+    );
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/capture_unauthed.rs

@@ -0,0 +1,83 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_capture_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    // POST /capture/set_config → 200 (authed)
+    let resp = authed(
+        client()
+            .post(format!(
+                "http://localhost:{port}/api/w/test-workspace/capture/set_config"
+            ))
+            .json(&json!({
+                "trigger_kind": "webhook",
+                "path": "u/test-user/test_capture",
+                "is_flow": false
+            })),
+    )
+    .send()
+    .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /capture/set_config");
+
+    // GET /capture/list/{...} → 200 (authed)
+    let resp = authed(client().get(format!(
+        "http://localhost:{port}/api/w/test-workspace/capture/list/script/u/test-user/test_capture"
+    )))
+    .send()
+    .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(
+        status,
+        &body,
+        "GET /capture/list/script/u/test-user/test_capture",
+    );
+
+    // POST /capture/ping_config/{trigger_kind}/{runnable_kind}/{*path} → 200
+    let resp = authed(client().post(format!(
+        "http://localhost:{port}/api/w/test-workspace/capture/ping_config/webhook/script/u/test-user/test_capture"
+    )))
+    .send()
+    .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /capture/ping_config",
+    );
+
+    // GET /capture/get_configs/{runnable_kind}/{*path} → 200
+    let resp = authed(client().get(format!(
+        "http://localhost:{port}/api/w/test-workspace/capture/get_configs/script/u/test-user/test_capture"
+    )))
+    .send()
+    .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /capture/get_configs",
+    );
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/concurrency_groups.rs

@@ -0,0 +1,48 @@
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_concurrency_groups_2xx(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    let resp = authed(client().get(format!(
+        "http://localhost:{port}/api/concurrency_groups/list"
+    )))
+    .send()
+    .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /api/concurrency_groups/list",
+    );
+
+    let resp = authed(client().get(format!(
+        "http://localhost:{port}/api/w/test-workspace/concurrency_groups/list_jobs"
+    )))
+    .send()
+    .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /api/w/test-workspace/concurrency_groups/list_jobs",
+    );
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/favorites.rs

@@ -0,0 +1,72 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_favorites_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let ws = format!("http://localhost:{port}/api/w/test-workspace");
+
+    // Setup: create a script to favorite
+    let resp = authed(client().post(format!("{ws}/scripts/create")))
+        .json(&json!({
+            "path": "u/test-user/test_fav_script",
+            "summary": "test",
+            "description": "",
+            "content": "export function main() { return 1; }",
+            "language": "deno",
+            "schema": {
+                "$schema": "https://json-schema.org/draft/2020-12/schema",
+                "type": "object",
+                "properties": {},
+                "required": []
+            }
+        }))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /scripts/create (setup)");
+
+    let fav_body = json!({
+        "favorite_kind": "script",
+        "path": "u/test-user/test_fav_script"
+    });
+
+    // POST /favorites/star → 200
+    let resp = authed(client().post(format!("{ws}/favorites/star")))
+        .json(&fav_body)
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /favorites/star");
+
+    // POST /favorites/unstar → 200
+    let resp = authed(client().post(format!("{ws}/favorites/unstar")))
+        .json(&fav_body)
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /favorites/unstar");
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/folder_history.rs

@@ -0,0 +1,51 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_folder_history_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    // Create a folder first
+    let resp = authed(
+        client()
+            .post(format!(
+                "http://localhost:{port}/api/w/test-workspace/folders/create"
+            ))
+            .json(&json!({"name": "test_hist_folder", "owners": ["u/test-user"]})),
+    )
+    .send()
+    .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /folders/create");
+
+    // GET /folders_history/get/{folder} → 200 (empty array)
+    let resp = authed(client().get(format!(
+        "http://localhost:{port}/api/w/test-workspace/folders_history/get/test_hist_folder"
+    )))
+    .send()
+    .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "GET /folders_history/get/test_hist_folder");
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/granular_acls.rs

@@ -0,0 +1,54 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_granular_acls_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/acls");
+
+    // GET /acls/get/group_/all → 200
+    let resp = authed(client().get(format!("{base}/get/group_/all")))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "GET /acls/get/group_/all");
+
+    // POST /acls/add/group_/all → 200
+    let resp = authed(client().post(format!("{base}/add/group_/all")))
+        .json(&json!({"owner": "u/test-user-2", "write": true}))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /acls/add/group_/all");
+
+    // POST /acls/remove/group_/all → 200
+    let resp = authed(client().post(format!("{base}/remove/group_/all")))
+        .json(&json!({"owner": "u/test-user-2"}))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /acls/remove/group_/all");
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/group_history.rs

@@ -0,0 +1,32 @@
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_group_history_2xx(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/groups_history");
+
+    let resp = authed(client().get(format!("{base}/get/all")))
+        .send()
+        .await?;
+    assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get/all");
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/health.rs

@@ -0,0 +1,41 @@
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_health_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/health");
+
+    // GET /health/status → 200 (no auth required)
+    let resp = client().get(format!("{base}/status")).send().await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "GET /health/status");
+
+    // GET /health/detailed → 200 (authed)
+    let resp = authed(client().get(format!("{base}/detailed")))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "GET /health/detailed");
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/inputs.rs

@@ -0,0 +1,76 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_inputs_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/inputs");
+
+    // GET /history with fake runnable → 200 empty array
+    let resp = authed(client().get(format!(
+        "{base}/history?runnable_id=u/test-user/test&runnable_type=ScriptPath"
+    )))
+    .send()
+    .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "GET /inputs/history");
+
+    // GET /list with fake runnable → 200 empty array
+    let resp = authed(client().get(format!(
+        "{base}/list?runnable_id=u/test-user/test&runnable_type=ScriptPath"
+    )))
+    .send()
+    .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "GET /inputs/list");
+
+    // POST /create → 200, returns UUID
+    let resp = authed(client().post(format!(
+        "{base}/create?runnable_id=u/test-user/test&runnable_type=ScriptPath"
+    )))
+    .json(&json!({"name": "test_input", "args": {}}))
+    .send()
+    .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /inputs/create");
+    let input_id: String = serde_json::from_str(&body)?;
+
+    // GET /{id}/args → 200
+    let resp = authed(client().get(format!("{base}/{input_id}/args")))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "GET /inputs/{id}/args");
+
+    // POST /delete/{id} → 200
+    let resp = authed(client().post(format!("{base}/delete/{input_id}")))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_2xx(status, &body, "POST /inputs/delete/{id}");
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/job_metrics.rs

@@ -0,0 +1,59 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+const FAKE_UUID: &str = "00000000-0000-0000-0000-000000000000";
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_job_metrics_2xx(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/job_metrics");
+
+    let resp = authed(client().post(format!("{base}/get/{FAKE_UUID}")))
+        .json(&json!({}))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /get/{id}",
+    );
+
+    let resp = authed(client().post(format!("{base}/set_progress/{FAKE_UUID}")))
+        .json(&json!({"percent": 50}))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /set_progress/{id}",
+    );
+
+    let resp = authed(client().get(format!("{base}/get_progress/{FAKE_UUID}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /get_progress/{id}",
+    );
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/jobs_authed.rs

@@ -0,0 +1,309 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use uuid::Uuid;
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+fn assert_route_reachable(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        status != 404 || !body.is_empty(),
+        "Router-level 404 for {endpoint}",
+    );
+}
+
+async fn insert_completed_job(db: &Pool<Postgres>) -> Uuid {
+    let id = Uuid::new_v4();
+    sqlx::query(
+        "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, args)
+         VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'script', 'deno', '{}'::jsonb)",
+    )
+    .bind(id)
+    .execute(db)
+    .await
+    .unwrap();
+    sqlx::query(
+        "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, result, status)
+         VALUES ($1, 'test-workspace', 100, '42'::jsonb, 'success')",
+    )
+    .bind(id)
+    .execute(db)
+    .await
+    .unwrap();
+    id
+}
+
+#[allow(dead_code)]
+async fn create_script(port: u16) -> String {
+    let base = format!("http://localhost:{port}/api/w/test-workspace/scripts");
+    let resp = authed(client().post(format!("{base}/create")))
+        .json(&json!({
+            "path": "u/test-user/test_job_script",
+            "summary": "test",
+            "description": "",
+            "content": "export function main() { return 42; }",
+            "language": "deno",
+            "schema": {
+                "$schema": "https://json-schema.org/draft/2020-12/schema",
+                "type": "object",
+                "properties": {},
+                "required": []
+            }
+        }))
+        .send()
+        .await
+        .unwrap();
+    assert!(
+        resp.status().is_success(),
+        "create script: {}",
+        resp.status()
+    );
+    "u/test-user/test_job_script".to_string()
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_jobs_authed_list_and_count(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/jobs");
+
+    // --- List/count endpoints (2xx with empty results) ---
+
+    let resp = authed(client().get(format!("{base}/list"))).send().await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/list",
+    );
+
+    let resp = authed(client().get(format!("{base}/queue/list")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/queue/list",
+    );
+
+    let resp = authed(client().get(format!("{base}/queue/count")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/queue/count",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/list")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/completed/list",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/count")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/completed/count",
+    );
+
+    // --- Global endpoints ---
+
+    let resp = client()
+        .get(format!("http://localhost:{port}/api/jobs/db_clock"))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/db_clock",
+    );
+
+    let resp = authed(client().get(format!(
+        "http://localhost:{port}/api/jobs/completed/count_by_tag"
+    )))
+    .send()
+    .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/completed/count_by_tag",
+    );
+
+    Ok(())
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_jobs_authed_completed_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/jobs");
+
+    let job_id = insert_completed_job(&db).await;
+
+    let resp = authed(client().get(format!("{base}/completed/get/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/completed/get",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/get_result/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/completed/get_result",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/get_result_maybe/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/completed/get_result_maybe",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/get_timing/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/completed/get_timing",
+    );
+
+    Ok(())
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_jobs_authed_run_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/jobs");
+
+    // Run preview — no pre-existing script needed
+    let resp = authed(client().post(format!("{base}/run/preview")))
+        .json(&json!({
+            "content": "export function main() { return 1; }",
+            "language": "deno",
+            "args": {}
+        }))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /jobs/run/preview",
+    );
+
+    // Run preview flow
+    let resp = authed(client().post(format!("{base}/run/preview_flow")))
+        .json(&json!({
+            "value": {"modules": []},
+            "args": {}
+        }))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /jobs/run/preview_flow",
+    );
+
+    Ok(())
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_jobs_authed_reachability(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/jobs");
+    let fake = Uuid::nil();
+
+    // These need complex runtime but should hit the handler (not 404)
+
+    let resp = authed(client().post(format!("{base}/flow/resume/{fake}")))
+        .json(&json!({}))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /jobs/flow/resume",
+    );
+
+    let resp = authed(client().get(format!("{base}/job_signature/{fake}/1")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/job_signature",
+    );
+
+    let resp = authed(client().get(format!("{base}/resume_urls/{fake}/1")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/resume_urls",
+    );
+
+    let resp = authed(client().get(format!("{base}/result_by_id/{fake}/step1")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /jobs/result_by_id",
+    );
+
+    let resp = authed(client().post(format!("{base}/restart/f/{fake}")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /jobs/restart/f",
+    );
+
+    let resp = authed(client().post(format!("{base}/run/workflow_as_code/{fake}/main")))
+        .json(&json!({}))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /jobs/run/workflow_as_code",
+    );
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/jobs_unauthed.rs

@@ -0,0 +1,493 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use uuid::Uuid;
+use windmill_common::variables::generate_approval_token;
+use windmill_test_utils::*;
+
+fn client() -> reqwest::Client {
+    reqwest::Client::new()
+}
+
+fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    builder.header("Authorization", "Bearer SECRET_TOKEN")
+}
+
+fn assert_2xx(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        (200..300).contains(&status),
+        "{endpoint} returned {status}: {body}",
+    );
+}
+
+/// Insert a minimal completed job directly into the database for testing.
+async fn insert_completed_job(db: &Pool<Postgres>) -> Uuid {
+    let id = Uuid::new_v4();
+    sqlx::query(
+        "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, args)
+         VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'script', 'deno', '{}'::jsonb)",
+    )
+    .bind(id)
+    .execute(db)
+    .await
+    .unwrap();
+
+    sqlx::query(
+        "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, result, status)
+         VALUES ($1, 'test-workspace', 100, '42'::jsonb, 'success')",
+    )
+    .bind(id)
+    .execute(db)
+    .await
+    .unwrap();
+
+    id
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_jobs_unauthed_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/jobs_u");
+
+    let job_id = insert_completed_job(&db).await;
+
+    // --- No-data endpoints ---
+
+    let resp = authed(client().post(format!("{base}/queue/get_started_at_by_ids")))
+        .json(&json!([]))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /queue/get_started_at_by_ids",
+    );
+
+    // --- Completed job endpoints (unauthed service, with auth header) ---
+
+    let resp = authed(client().get(format!("{base}/get/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get");
+
+    let resp = authed(client().get(format!("{base}/get_logs/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get_logs");
+
+    let resp = authed(client().get(format!("{base}/get_completed_logs_tail/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /get_completed_logs_tail",
+    );
+
+    let resp = authed(client().get(format!("{base}/get_args/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get_args");
+
+    let resp = authed(client().get(format!("{base}/completed/get/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /completed/get",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/get_result/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /completed/get_result",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/get_result_maybe/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /completed/get_result_maybe",
+    );
+
+    let resp = authed(client().get(format!("{base}/completed/get_timing/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /completed/get_timing",
+    );
+
+    let resp = authed(client().get(format!("{base}/getupdate/{job_id}")))
+        .send()
+        .await?;
+    assert_2xx(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /getupdate",
+    );
+
+    Ok(())
+}
+
+const FAKE_UUID: &str = "00000000-0000-0000-0000-000000000000";
+const FAKE_SECRET: &str = "aabb";
+
+/// Reachability tests for endpoints that need complex runtime.
+/// These just verify the route matches (handler runs), not 2xx.
+fn assert_route_reachable(status: u16, body: &str, endpoint: &str) {
+    assert!(
+        status != 404 || !body.is_empty(),
+        "Router-level 404 for {endpoint}",
+    );
+}
+
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_jobs_unauthed_complex_reachability(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/jobs_u");
+
+    let resp = authed(client().get(format!("{base}/resume/{FAKE_UUID}/1/{FAKE_SECRET}")))
+        .send()
+        .await?;
+    assert_route_reachable(resp.status().as_u16(), &resp.text().await?, "GET /resume");
+
+    let resp = authed(client().post(format!("{base}/cancel/{FAKE_UUID}/1/{FAKE_SECRET}")))
+        .send()
+        .await?;
+    assert_route_reachable(resp.status().as_u16(), &resp.text().await?, "POST /cancel");
+
+    let resp = authed(client().get(format!("{base}/get_flow/{FAKE_UUID}/1/{FAKE_SECRET}")))
+        .send()
+        .await?;
+    assert_route_reachable(resp.status().as_u16(), &resp.text().await?, "GET /get_flow");
+
+    let resp = authed(client().post(format!("{base}/queue/cancel/{FAKE_UUID}")))
+        .json(&serde_json::json!({"reason": "test"}))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /queue/cancel",
+    );
+
+    let resp = authed(client().post(format!("{base}/queue/force_cancel/{FAKE_UUID}")))
+        .json(&serde_json::json!({"reason": "test"}))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /queue/force_cancel",
+    );
+
+    let resp = authed(client().post(format!("{base}/flow/resume_suspended/{FAKE_UUID}")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /flow/resume_suspended",
+    );
+
+    let resp = authed(client().get(format!("{base}/flow/approval_info/{FAKE_UUID}")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /flow/approval_info",
+    );
+
+    let resp = authed(client().get(format!("{base}/get_root_job_id/{FAKE_UUID}")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /get_root_job_id",
+    );
+
+    let resp = authed(client().get(format!("{base}/get_flow_debug_info/{FAKE_UUID}")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /get_flow_debug_info",
+    );
+
+    let resp = authed(client().get(format!("{base}/get_log_file/{FAKE_UUID}/test.txt")))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "GET /get_log_file",
+    );
+
+    let resp = authed(client().post(format!("{base}/queue/cancel_persistent/u/test-user/fake")))
+        .json(&serde_json::json!({"reason": "test"}))
+        .send()
+        .await?;
+    assert_route_reachable(
+        resp.status().as_u16(),
+        &resp.text().await?,
+        "POST /queue/cancel_persistent",
+    );
+
+    Ok(())
+}
+
+/// Build a minimal FlowValue JSON with a suspend step (with resume_form) followed by an identity
+/// step. The suspend step is at index 0, the identity step at index 1. After the suspend step
+/// completes, `flow_status.step` = 1 and `approval_step = 0` points to the form.
+fn flow_value_with_suspend_form() -> serde_json::Value {
+    json!({
+        "modules": [
+            {
+                "id": "a",
+                "value": {"type": "identity"},
+                "suspend": {
+                    "required_events": 1,
+                    "resume_form": {
+                        "schema": {
+                            "properties": {
+                                "reason": {"type": "string", "description": "Approval reason"}
+                            },
+                            "order": ["reason"]
+                        }
+                    }
+                }
+            },
+            {
+                "id": "b",
+                "value": {"type": "identity"}
+            }
+        ],
+        "same_worker": false
+    })
+}
+
+/// Build the flow_status JSON for a flow suspended at step 1 (step 0 completed with suspend).
+fn flow_status_suspended_at_step_1(step_job_id: Uuid) -> serde_json::Value {
+    json!({
+        "step": 1,
+        "modules": [
+            {"type": "Success", "id": "a", "job": step_job_id, "skipped": false},
+            {"type": "WaitingForEvents", "id": "b", "count": 1, "job": step_job_id}
+        ],
+        "failure_module": {
+            "parent_module": null,
+            "type": "WaitingForPriorSteps",
+            "id": "failure"
+        },
+        "cleanup_module": {"flow_jobs_to_clean": []}
+    })
+}
+
+/// Insert a v2_job_queue + v2_job_status pair (v2_job_status has FK to v2_job_queue).
+async fn insert_queue_and_status(
+    db: &Pool<Postgres>,
+    flow_id: Uuid,
+    flow_status: &serde_json::Value,
+) {
+    sqlx::query(
+        "INSERT INTO v2_job_queue (id, workspace_id, scheduled_for)
+         VALUES ($1, 'test-workspace', now())",
+    )
+    .bind(flow_id)
+    .execute(db)
+    .await
+    .unwrap();
+
+    sqlx::query(
+        "INSERT INTO v2_job_status (id, flow_status)
+         VALUES ($1, $2)",
+    )
+    .bind(flow_id)
+    .bind(flow_status)
+    .execute(db)
+    .await
+    .unwrap();
+}
+
+/// Insert a flow job with raw_flow stored in v2_job (RawFlow / FlowPreview path).
+async fn insert_suspended_flow_with_raw_flow(
+    db: &Pool<Postgres>,
+    raw_flow: &serde_json::Value,
+) -> Uuid {
+    let flow_id = Uuid::new_v4();
+    let step_job_id = Uuid::new_v4();
+    let flow_status = flow_status_suspended_at_step_1(step_job_id);
+
+    sqlx::query(
+        "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, raw_flow)
+         VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'flowpreview', 'flow', $2)",
+    )
+    .bind(flow_id)
+    .bind(raw_flow)
+    .execute(db)
+    .await
+    .unwrap();
+
+    insert_queue_and_status(db, flow_id, &flow_status).await;
+
+    flow_id
+}
+
+/// Insert a flow job WITHOUT raw_flow but with a matching flow_node entry (FlowNode path).
+async fn insert_suspended_flow_node(db: &Pool<Postgres>, flow_value: &serde_json::Value) -> Uuid {
+    let flow_id = Uuid::new_v4();
+    let step_job_id = Uuid::new_v4();
+    let flow_status = flow_status_suspended_at_step_1(step_job_id);
+
+    // Insert the flow_node entry first to get its id
+    let node_id: i64 = sqlx::query_scalar(
+        "INSERT INTO flow_node (workspace_id, hash, path, flow)
+         VALUES ('test-workspace', 12345, 'f/test/flow', $1)
+         RETURNING id",
+    )
+    .bind(flow_value)
+    .fetch_one(db)
+    .await
+    .unwrap();
+
+    // Insert the job with kind=flownode and runnable_id pointing to the flow_node
+    sqlx::query(
+        "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, runnable_id)
+         VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'flownode', 'flow', $2)",
+    )
+    .bind(flow_id)
+    .bind(node_id)
+    .execute(db)
+    .await
+    .unwrap();
+
+    insert_queue_and_status(db, flow_id, &flow_status).await;
+
+    flow_id
+}
+
+/// Insert a flow job WITHOUT raw_flow but with runnable_id pointing to flow_node,
+/// and NO flow_node entry — simulates the broken state before the fix.
+async fn insert_suspended_flow_node_without_node_entry(db: &Pool<Postgres>) -> Uuid {
+    let flow_id = Uuid::new_v4();
+    let step_job_id = Uuid::new_v4();
+    let flow_status = flow_status_suspended_at_step_1(step_job_id);
+
+    // Use a non-existent runnable_id — simulates FlowNode with raw_flow=NULL and no fallback
+    sqlx::query(
+        "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, runnable_id)
+         VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'flownode', 'flow', 99999999)",
+    )
+    .bind(flow_id)
+    .execute(db)
+    .await
+    .unwrap();
+
+    insert_queue_and_status(db, flow_id, &flow_status).await;
+
+    flow_id
+}
+
+async fn get_approval_info_response(
+    port: u16,
+    db: &Pool<Postgres>,
+    job_id: Uuid,
+) -> serde_json::Value {
+    let token = generate_approval_token("test-workspace", job_id, db)
+        .await
+        .unwrap();
+    let base = format!("http://localhost:{port}/api/w/test-workspace/jobs_u");
+    let resp = client()
+        .get(format!("{base}/flow/approval_info/{job_id}?token={token}"))
+        .send()
+        .await
+        .unwrap();
+    let status = resp.status().as_u16();
+    let body = resp.text().await.unwrap();
+    assert!(
+        (200..300).contains(&status),
+        "approval_info returned {status}: {body}",
+    );
+    serde_json::from_str(&body).unwrap()
+}
+
+/// Test: approval_info returns form_schema for a top-level suspend (raw_flow stored in v2_job).
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_approval_info_form_schema_from_raw_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    let flow_value = flow_value_with_suspend_form();
+    let flow_id = insert_suspended_flow_with_raw_flow(&db, &flow_value).await;
+
+    let info = get_approval_info_response(port, &db, flow_id).await;
+    assert!(
+        info.get("form_schema").is_some(),
+        "form_schema should be present for raw_flow path, got: {info}",
+    );
+
+    Ok(())
+}
+
+/// Test: approval_info returns form_schema for a FlowNode sub-flow (graph-based branch/loop).
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_approval_info_form_schema_from_flow_node(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    // We need a flow path to exist for the flow_node FK
+    sqlx::query(
+        "INSERT INTO flow (workspace_id, path, summary, description, versions, value, edited_by, edited_at, schema)
+         VALUES ('test-workspace', 'f/test/flow', '', '', '{}', '{}'::jsonb, 'test-user', now(), '{}'::jsonb)",
+    )
+    .execute(&db)
+    .await?;
+
+    let flow_value = flow_value_with_suspend_form();
+    let flow_id = insert_suspended_flow_node(&db, &flow_value).await;
+
+    let info = get_approval_info_response(port, &db, flow_id).await;
+    assert!(
+        info.get("form_schema").is_some(),
+        "form_schema should be present for flow_node path, got: {info}",
+    );
+
+    Ok(())
+}
+
+/// Test: approval_info returns no form_schema when FlowNode has no matching entry
+/// (simulates the pre-fix behavior where flow_node fallback didn't exist).
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_approval_info_no_form_when_flow_node_missing(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    let flow_id = insert_suspended_flow_node_without_node_entry(&db).await;
+
+    let info = get_approval_info_response(port, &db, flow_id).await;
+    assert!(
+        info.get("form_schema").is_none(),
+        "form_schema should be absent when no flow definition found, got: {info}",
+    );
+
+    Ok(())
+}