worker-batch-pull-write
41 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
hugocasa
|
bffa61e33f |
fix: dedicated worker dispatch, cross-workspace deps, UI improvements (#8689)
* feat: restore bun as default runtime for dedicated workers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: add context comment for bun dedicated worker nodejs migration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: dedicated worker dispatch for flows + add E2E tests - Add workspace_id prefix to dedicated worker map lookup keys - Update ee-repo-ref for dedicated worker path handling fix - Add spawn_test_worker_dedicated/in_test_worker_dedicated test helpers - Add 6 E2E tests for dedicated workers: - test_dedicated_flow_rawscript (regression for "Script not found" bug) - test_dedicated_flow_workspace_script - test_dedicated_flow_multiple_steps - test_dedicated_standalone_script - test_dedicated_runner_group - test_dedicated_flow_runners - Add dedicated_flows.sql fixture with scripts, flows, and worker config Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: always run dependency job for dedicated worker scripts When a script with dedicated_worker=true is deployed with a pre-computed lock (e.g. via wmill sync push), no dependency job was created, so the dedicated worker never detected the update and kept running the old version. Now dedicated worker scripts always generate a dependency job regardless of whether a lock is provided. The dependency job runs on the dedicated worker and triggers a restart so it picks up the new script version. Fixes #8638 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use serial_test for dedicated worker tests to avoid WORKER_CONFIG races Dedicated worker tests need non-default worker tags in the global WORKER_CONFIG. When run in parallel (CI uses --test-threads=10), multiple tests clobber each other's config. Use #[serial] to ensure dedicated worker tests run sequentially. Also load worker config from DB via load_worker_config() instead of manually setting WORKER_CONFIG fields, ensuring consistency with the monitor's reload path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: nodejs dedicated worker script_path shadowing + add multi-language E2E tests Fix script_path shadowing in bun_executor nodejs branch where the wrapper file path was passed to handle_dedicated_process instead of the logical path, causing "Script not found" for all //nodejs dedicated workers. Add E2E tests for dedicated flows in all supported languages: - test_dedicated_flow_deno - test_dedicated_flow_python - test_dedicated_flow_bunnative (V8 PrewarmedIsolate path) - test_dedicated_flow_bun_nodejs (//nodejs annotation) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: simplify dedicated worker dispatch + add serialization and E2E tests - Unified lookup: always use {workspace}:{runnable_path} for dedicated worker dispatch, replacing the flow_step_id iteration approach - Added serialization_semaphore parameter to executor start_worker fns - Added E2E tests: cross-workspace isolation, conflicting flow step IDs, preprocessor on dedicated worker - Added workspace field to RunJob for cross-workspace test support Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: cross-workspace workspace dependencies on workers page Add two new instance-level endpoints to the configs router: - GET /configs/list_all_workspace_dependencies - GET /configs/list_all_dedicated_with_deps Both require devops role and return data across all workspaces, enabling the workers page to show a consistent view of which workspace dependencies exist regardless of which workspace the user is browsing. Update DedicatedWorkersSelector to use the new cross-workspace endpoints with fallback to per-workspace calls for non-devops users. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to include dedicated worker lookup simplification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: use branch name for ee-repo-ref (CI can't fetch by SHA from non-default branch) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update ee-repo-ref.txt with new reference * sqlx * fix: revert serialization semaphore, multi-workspace picker, dep conflict warnings - Remove serialization_semaphore from executor start_worker signatures - Remove serialization test and fixtures - Fix DedicatedWorkersSelector to preserve tags from other workspaces when toggling in the picker - Track workspace deps per-workspace for conflict detection - Show warning when dep exists in another workspace but not the script's - Group runner groups per-workspace to prevent cross-workspace merging - Add workspace to dep badge link URL Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: simplify exec protocol — execd: for single-script, exec: for runner groups Add execd:/execd_preprocess: commands to bun/deno/python wrappers for single-script dedicated workers (no path needed). Runner groups keep exec:/exec_preprocess: with path for multi-script disambiguation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add unit tests for execd:/exec: wrapper protocol Verify generate_multi_script_wrapper produces both execd: (single-script) and exec: (runner group) protocol handlers, including preprocessor variants. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update commit reference in ee-repo-ref.txt * fix: remove beta badge from squash loop, keep tooltip Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update protocol tests to use execd: for single-script wrappers Deno and bun single-script protocol tests now send execd:{args} instead of exec:{path}:{args}, matching the updated wrapper protocol. Multi-script (runner group) tests continue to use exec:{path}:{args}. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove unused TEST_SCRIPT_PATH in deno protocol tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: review feedback — down migration, push_as workspace, UI improvements - Use regexp_replace in down migration for positional accuracy - Fix push_as() to use self.workspace_id instead of hardcoded value - Remove per-workspace API fallbacks, use cross-workspace endpoints only - Skip devops-only API calls when user is not devops (disabled prop) - Fix duplicate key error for cross-workspace runner groups - Add workspace to RunnerGroup for unique keying - Reuse tagRow snippet for standalone items with expand/collapse - Fix picker alignment: remove empty column for non-expandable items Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: comprehensive dedicated worker test coverage, fix Python execd_preprocess - Add Python execd_preprocess: handler (was missing for single-script dedicated workers) - Add 10 E2E tests: flow+standalone conflict, mixed lang fallback, unsupported lang flow runners, python runner group, bun/python/deno/bunnative preprocessors, runner group preprocessors, branchone flow - Add 4 Python unit tests for execd:/execd_preprocess: protocol - Update EE ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: review feedback — migration escaping, deno try/catch, loadRunnables guard - Down migration: use E'...' so \n matches actual newlines - Up migration: anchor regex with ^ to avoid mid-content matches - Deno execd_preprocess: move JSON.parse inside try/catch - DedicatedWorkersSelector: skip devops-only API calls when disabled Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add dedicated worker relative import tests for bun and python Verifies that build_loader's CURRENT_PATH correctly resolves workspace- relative imports when running on a dedicated worker subprocess. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: dedicated worker dispatch for nested flow structures (branches/loops) - Add extract_flow_root() to strip nesting segments from runnable_path - Dispatch uses flow_root/flow_step_id for nested paths, runnable_path for flat paths — deterministic, O(1) - Fix assert_ran_on_dedicated_worker to BFS all descendants - Fix python mode labels (python vs python3 for runner groups) - Add tests: simple forloop, multi-step forloop, whileloop, branchall, nested branch-in-loop, mixed lang fallback, unsupported lang runners Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: fix ee-repo-ref SHA Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: hide picker and skip API calls for read-only users, hide empty runner badge - Hide "Add more scripts/flows" section when disabled (read-only) - Skip per-runnable API calls (getScriptByPath, getFlowByPath) for disabled users — just show path info - Hide "0 runners" badge on flows with no eligible steps Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 9422b189762ae27edfc346541ae668a4ad728325 This commit updates the EE repository reference after PR #503 was merged in windmill-ee-private. Previous ee-repo-ref: 4c6ba214bfc23fff05d1dc3200ac59e650af3f4f New ee-repo-ref: 9422b189762ae27edfc346541ae668a4ad728325 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasa
|
61a867f086 |
Revert "feat: restore bun for dedicated workers, fix dispatch & serialization, cross-workspace deps (#8645)" (#8687)
This reverts commit
|
||
|
hugocasa
|
619ebb65ce |
feat: restore bun for dedicated workers, fix dispatch & serialization, cross-workspace deps (#8645)
* feat: restore bun as default runtime for dedicated workers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: add context comment for bun dedicated worker nodejs migration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: dedicated worker dispatch for flows + add E2E tests - Add workspace_id prefix to dedicated worker map lookup keys - Update ee-repo-ref for dedicated worker path handling fix - Add spawn_test_worker_dedicated/in_test_worker_dedicated test helpers - Add 6 E2E tests for dedicated workers: - test_dedicated_flow_rawscript (regression for "Script not found" bug) - test_dedicated_flow_workspace_script - test_dedicated_flow_multiple_steps - test_dedicated_standalone_script - test_dedicated_runner_group - test_dedicated_flow_runners - Add dedicated_flows.sql fixture with scripts, flows, and worker config Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: always run dependency job for dedicated worker scripts When a script with dedicated_worker=true is deployed with a pre-computed lock (e.g. via wmill sync push), no dependency job was created, so the dedicated worker never detected the update and kept running the old version. Now dedicated worker scripts always generate a dependency job regardless of whether a lock is provided. The dependency job runs on the dedicated worker and triggers a restart so it picks up the new script version. Fixes #8638 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use serial_test for dedicated worker tests to avoid WORKER_CONFIG races Dedicated worker tests need non-default worker tags in the global WORKER_CONFIG. When run in parallel (CI uses --test-threads=10), multiple tests clobber each other's config. Use #[serial] to ensure dedicated worker tests run sequentially. Also load worker config from DB via load_worker_config() instead of manually setting WORKER_CONFIG fields, ensuring consistency with the monitor's reload path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: nodejs dedicated worker script_path shadowing + add multi-language E2E tests Fix script_path shadowing in bun_executor nodejs branch where the wrapper file path was passed to handle_dedicated_process instead of the logical path, causing "Script not found" for all //nodejs dedicated workers. Add E2E tests for dedicated flows in all supported languages: - test_dedicated_flow_deno - test_dedicated_flow_python - test_dedicated_flow_bunnative (V8 PrewarmedIsolate path) - test_dedicated_flow_bun_nodejs (//nodejs annotation) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: simplify dedicated worker dispatch + add serialization and E2E tests - Unified lookup: always use {workspace}:{runnable_path} for dedicated worker dispatch, replacing the flow_step_id iteration approach - Added serialization_semaphore parameter to executor start_worker fns - Added E2E tests: cross-workspace isolation, conflicting flow step IDs, preprocessor on dedicated worker - Added workspace field to RunJob for cross-workspace test support Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: cross-workspace workspace dependencies on workers page Add two new instance-level endpoints to the configs router: - GET /configs/list_all_workspace_dependencies - GET /configs/list_all_dedicated_with_deps Both require devops role and return data across all workspaces, enabling the workers page to show a consistent view of which workspace dependencies exist regardless of which workspace the user is browsing. Update DedicatedWorkersSelector to use the new cross-workspace endpoints with fallback to per-workspace calls for non-devops users. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to include dedicated worker lookup simplification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: use branch name for ee-repo-ref (CI can't fetch by SHA from non-default branch) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update ee-repo-ref.txt with new reference * sqlx * fix: revert serialization semaphore, multi-workspace picker, dep conflict warnings - Remove serialization_semaphore from executor start_worker signatures - Remove serialization test and fixtures - Fix DedicatedWorkersSelector to preserve tags from other workspaces when toggling in the picker - Track workspace deps per-workspace for conflict detection - Show warning when dep exists in another workspace but not the script's - Group runner groups per-workspace to prevent cross-workspace merging - Add workspace to dep badge link URL Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: simplify exec protocol — execd: for single-script, exec: for runner groups Add execd:/execd_preprocess: commands to bun/deno/python wrappers for single-script dedicated workers (no path needed). Runner groups keep exec:/exec_preprocess: with path for multi-script disambiguation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add unit tests for execd:/exec: wrapper protocol Verify generate_multi_script_wrapper produces both execd: (single-script) and exec: (runner group) protocol handlers, including preprocessor variants. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update commit reference in ee-repo-ref.txt * fix: remove beta badge from squash loop, keep tooltip Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update protocol tests to use execd: for single-script wrappers Deno and bun single-script protocol tests now send execd:{args} instead of exec:{path}:{args}, matching the updated wrapper protocol. Multi-script (runner group) tests continue to use exec:{path}:{args}. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove unused TEST_SCRIPT_PATH in deno protocol tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
efb4a27d51 |
fix: replace email with permissioned_as for triggers/schedules (#8439)
* refactor: replace email with permissioned_as for triggers/schedules
Add a new `permissioned_as` column (format: `u/{username}`, `g/{group}`,
or raw email) to all trigger tables and schedule. This value is used
directly for job permission checks, removing the need for email lookups
when creating/updating triggers.
- Migration: add permissioned_as to all 9 trigger tables + schedule,
drop email from trigger tables (schedule keeps it for backwards compat)
- Backend: resolve_email() (async, DB) -> resolve_permissioned_as() (sync)
- Email cache: get_email_from_permissioned_as() with quick_cache for
places that still need email (fetch_api_authed, schedule backwards compat)
- Frontend: rename email/preserve_email -> permissioned_as/preserve_permissioned_as
in deploy data and OpenAPI schemas
- Tests updated for new field names and u/{username} format
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix sqlx/build
* update ee ref
* refactor: simplify resolve_edited_by to always use authed username
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix compile + migration
* update ref
* test: add trigger trait method tests for permissioned_as queries
Add tests that call TriggerCrud and Listener trait methods directly
to verify dynamic SQL correctly references the permissioned_as column.
Covers get_trigger_by_path, list_triggers, set_trigger_mode, and
fetch_enabled_unlistened_triggers for all trigger types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* update sqlx
* fix: use permissioned_as directly for schedules and fix audit RLS for groups
- Schedule: permissioned_as only set on create, not on edit/set_enabled
- Schedule: stop reading email column, use get_email_from_permissioned_as
- Triggers: use fetch_api_authed_from_permissioned_as instead of edited_by
- Triggers: rename listener fields for clarity (username -> edited_by)
- Fix audit author username for group permissioned_as (g/test -> group-test)
to match session.user, preventing RLS policy violations on audit_partitioned
- OpenAPI: remove permissioned_as/preserve_permissioned_as from EditSchedule
- Add backwards-compat comments for schedule email writes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate system prompts for permissioned_as field
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix build
* refactor: generalize onBehalfOf naming, add permissioned_as to EditSchedule
- Frontend: rename onBehalfOfPermissionedAs -> onBehalfOf with comments
explaining it carries emails for flows/scripts and permissioned_as for
triggers/schedules
- Frontend: rename getOnBehalfOfEmail -> getOnBehalfOf,
getOnBehalfOfPermissionedAsForDeploy -> getOnBehalfOfForDeploy,
customOnBehalfOfEmails -> customOnBehalfOf
- Backend: add optional permissioned_as/preserve_permissioned_as to
EditSchedule with COALESCE (only updates when provided)
- Backend: add on_behalf_of audit log for schedule edit
- Backend: remove unused resolve_on_behalf_of_permissioned_as
- Tests: remove email assertions from schedule update test (email is
just backwards compat, only permissioned_as matters)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: preserve email column when permissioned_as is preserved on schedule edit
Derive email from the preserved permissioned_as via cache lookup instead
of always writing authed.email. This keeps the email column consistent
with the old behavior for backwards compat with old workers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update deploy UI labels from "edited by" to "run as" for triggers
Triggers now use permissioned_as (not edited_by) for permissions, so
update the deploy UI wording to reflect this. Also update wm_deployers
group description to mention schedules and permissioned_as.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use u/username format for custom trigger/schedule deploy selection
When picking a custom user for trigger/schedule deployment, store
u/${username} (permissioned_as format) instead of the email. Flows/scripts
continue to use email format for on_behalf_of_email.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: show u/username format for "me" option in trigger deploy selector
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: simplify OnBehalfOfSelector to return the right format per kind
OnBehalfOfSelector now handles the email vs permissioned_as format
internally based on kind:
- triggers: returns u/username, displays u/username in all options
- flows/scripts/apps: returns email, displays username
The onSelect callback now takes (choice, value?) where value is already
in the correct format. Parent components just store it directly without
needing to know about the format difference.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: always show u/username format in OnBehalfOfSelector for all kinds
Display is now consistent: all kinds show u/username in the selector.
The returned value still differs (email for flows/scripts, u/username
for triggers) since the backend APIs expect different formats.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: replace email with permissioned_as in http_trigger test insert
The email column was dropped from trigger tables in the migration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: review fixes — migration, app policy, capture cleanup, naming
- Migration: remove DEFAULT '', use nullable → populate → SET NOT NULL
- App policy: set both on_behalf_of and on_behalf_of_email for all choices
- OnBehalfOfSelector: return OnBehalfOfDetails {email, permissionedAs} instead of ambiguous value
- Remove unused email field from Capture struct and query
- Rename getSourceEmail/getTargetEmail → getSourceOnBehalfOf/getTargetOnBehalfOf
- Rename test functions from preserve_email to preserve_permissioned_as
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add permissioned_as to all test schedule INSERTs
Since the migration no longer uses DEFAULT '', all INSERTs must
explicitly provide permissioned_as. Updated test fixtures and
schedule_push tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: strip permissioned_as from exports/sync, fix OpenAPI required field
- Add permissioned_as to workspace export strip list (like edited_by)
- Add permissioned_as to CLI TriggerFile Omit list
- Fix TriggerExtraProperty.required: email → permissioned_as
- Regenerate frontend and CLI types
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove accidentally committed generated files
These directories are gitignored and should not be tracked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate system prompts for permissioned_as schema changes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove permissioned_as from CLI TriggerFile Omit list
Already stripped in workspace export, no need to also omit from the type.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: optimize email cache key and revert TriggerFile Omit change
- Use single concatenated string for cache key instead of (String, String) tuple
- Remove permissioned_as from CLI TriggerFile Omit (already stripped in export)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: zero-allocation email cache lookups using Equivalent trait
Use a borrowed EmailCacheKey(&str, &str) for cache lookups via
quick_cache's Equivalent support. Only allocates (String, String)
on cache miss for insert. This is called on every trigger fire
and schedule push.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add permissioned_as to Schedule required fields in OpenAPI spec
The backend always returns permissioned_as (non-optional String),
so the schema should reflect that.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: handle group- prefix in migration UPDATE statements
edited_by can be 'group-{name}' for group-owned triggers/schedules.
The migration now correctly maps these to 'g/{name}' format instead
of incorrectly producing 'u/group-{name}'.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "fix: handle group- prefix in migration UPDATE statements"
This reverts commit
|
||
|
Ruben Fiszel
|
f2be625348 |
feat: store hashed tokens instead of plaintext (#8217)
* feat: store hashed tokens in the token table instead of plaintext
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address review issues in token hash migration
- Update all base.sql fixtures to include token_hash/token_prefix columns
- Keep plaintext token for webhook tokens (needed for URL reconstruction)
- Restore get_token_by_prefix to query DB for webhook tokens
- Fix down migration to delete NULL-token rows before restoring NOT NULL
- Update parser fixture standalone schema
- Update EE dedicated_worker_ee.rs to use token_hash/token_prefix
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: restore sqlx offline cache (only add new query files)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: keep writing plaintext token column for backward compat
Write to token column alongside token_hash until MIN_VERSION_SUPPORTS_TOKEN_HASH
(1.649.0) is reached. This ensures older workers can still authenticate
during rolling upgrades. Remove the separate UPDATE in new_webhook_token
since create_token_internal now writes plaintext directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: branch on MIN_VERSION to write plaintext token or null
Check MIN_VERSION_SUPPORTS_TOKEN_HASH at runtime: write plaintext to
token column while old workers exist, switch to NULL once all workers
are >= 1.649.0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: set MIN_VERSION_SUPPORTS_TOKEN_HASH to 1.650.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use token_hash for email lookup and expiry notifications
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: rotate webhook tokens instead of recovering plaintext from DB
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: use token_hash for native trigger token lookups and deletes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* sqlx
* refactor: drop webhook_token_prefix from native_trigger table
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: backward compat for token rotation and make webhook_token_hash NOT NULL
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: prevent panic on short superadmin secret token prefix
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: prevent panic on short superadmin secret token prefix
Replace all `token[0..TOKEN_PREFIX_LEN]` slicing with
`token.get(..TOKEN_PREFIX_LEN).unwrap_or(token)` to prevent
panics when a token shorter than 10 chars is provided (e.g.
malformed Authorization header, short superadmin secret).
Co-authored-by: hugocasa <hugocasa@users.noreply.github.com>
* fix: prevent panic on short token prefix slicing
Replace all `token[0..TOKEN_PREFIX_LEN]` with safe
`token.get(..TOKEN_PREFIX_LEN).unwrap_or(token)` to prevent panics
on malformed tokens shorter than 10 characters.
Co-authored-by: hugocasa <hugocasa@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Revert "fix: prevent panic on short superadmin secret token prefix"
This reverts commit
|
||
|
hugocasa
|
dc0e59f432 |
feat: add preprocessor support for dedicated workers and bunnative scripts (#8284)
* feat: add preprocessor support for dedicated workers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update ee-repo-ref.txt Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: extract transform_and_run helper in python dedicated wrapper Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add preprocessor support for bunnative scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: resolve unused postprocessor variable in python wrapper Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: add workflow_dispatch trigger to backend integration tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: bunnative fixture lock format and PrewarmedIsolate::spawn callers Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: update generate_dedicated_worker_wrapper callers in bun_jobs test Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: use non-dedicated workers in preprocessor integration tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: prewarm preprocessor isolate for bunnative dedicated workers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: flatten bunnative dedicated worker preprocessing into single result path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use labeled block instead of async block for EE compatibility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update commit reference in ee-repo-ref.txt * chore: update ee-repo-ref to e36945b987f7904fa984181baf3124e7b2722bd1 This commit updates the EE repository reference after PR #445 was merged in windmill-ee-private. Previous ee-repo-ref: 8a2625833452aadb8907242bf502b24ca2dffd73 New ee-repo-ref: e36945b987f7904fa984181baf3124e7b2722bd1 Automated by sync-ee-ref workflow. * Fix merge conflict in ee-repo-ref.txt Resolve merge conflict in ee-repo-ref.txt --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Pyra
|
baf2bcf14d |
feat: make WM_END_USER_EMAIL display users from different workspaces (#8208)
Signed-off-by: pyranota <pyra@duck.com> |
||
|
wendrul
|
7ac93f6ee3 | feat: option to preserve on_behalf_of and edited_by for admins and users in the new wm_deployers group (#8079) | ||
|
Ruben Fiszel
|
4fedfdfd11 |
feat(cli): add consistent get/list/new subcommands for all item types (#8047)
* feat(cli): add consistent get/list/new subcommands for all item types Make the CLI consistent so every item type (script, flow, app, resource, resource-type, variable, schedule, folder, trigger) supports get/list/new subcommands, enabling the CLI to be used as a full API client in bash scripts with jq piping. - Add --json flag to all list commands for machine-readable output - Register explicit "list" subcommand alongside default action - Add "get <path> [--json]" subcommand to fetch single items from API - Rename "bootstrap" to "new" for script/flow, keep "bootstrap" as alias - Add "new" subcommand for resource, resource-type, variable, schedule, folder, and trigger to create local template YAML files - Update cli-commands skill documentation for wmill init - Add integration tests for all new commands Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * feat: install wmill CLI in Docker images and use it for bash variable/resource access - Install windmill-cli via bun in all Dockerfiles that include bun - DockerfileCli: switch from node:slim to oven/bun:slim - CLI: auto-configure from WM_WORKSPACE/WM_TOKEN/BASE_INTERNAL_URL env vars as last-resort fallback when no workspace is configured - Frontend: replace curl-based bash snippets with wmill variable/resource get - Add backend integration tests for wmill CLI in bash scripts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): install windmill-cli in backend test workflow Ensures wmill is available on PATH for bash integration tests that use `wmill variable get` and `wmill resource get`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(cli): replace @std/* Deno dependencies with Node.js equivalents Replace @std/log with a lightweight custom logger (core/log.ts), @std/path with node:path, and @std/yaml with the yaml npm package. Also fix process hang on exit, add --node option to install_dev.sh, and add missing hasRequiredPermissions to NpmProvider. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * all * refactor(cli): replace @ayonli/jsext and @std/encoding with lightweight alternatives Replace @ayonli/jsext (8.4MB) with tar-stream (32kB) for tar creation, replace @std/encoding with Node.js Buffer.toString("hex"), and fix @windmill-labs/shared-utils to use direct npm instead of JSR mirror. Also resolve merge conflicts in sync.ts and fix pre-existing type errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): use singleQuote YAML output and pass yamlOptions in gitsync pull The yaml library defaults to double quotes, but the codebase (and tests) expect single-quoted strings. Add singleQuote: true to yamlOptions and pass yamlOptions to gitsync-settings pull writeFile calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * fix(cli): address code review feedback - Install CLI from source in backend tests instead of npm - Fix script bootstrap catch block to re-throw "File already exists" - Add type-safe local variable after trigger kind validation - Use created_by instead of policy.on_behalf_of for app get output - Note --kind is recommended for faster trigger lookup in help text - Document node symlink purpose in Dockerfiles Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): use /usr/bin for wmill wrapper to ensure it's in PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): install wmill to ~/.local/bin to avoid permission issues Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(backend): switch to Blacksmith runner and add cargo caching - Switch from ubicloud-standard-16 to blacksmith-16vcpu-ubuntu-2404 for faster NVMe-backed builds - Add stickydisk for cargo target directory (persistent NVMe cache across runs) - Add cache for cargo registry and git dependencies - Upgrade DuckDB FFI cache from actions/cache@v3 to useblacksmith/cache@v1 - Enable CARGO_INCREMENTAL=1 to benefit from persistent target cache Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix ci --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ff70a4e9d1 |
fix: parse Python datetime.datetime and datetime.date type annotations (#7856)
* fix: parse Python datetime.datetime and datetime.date type annotations correctly The Python parser only matched ExprKind::Name for type annotations, so `datetime.datetime` (an Attribute expression) silently fell through to Typ::Unknown and no datetime picker was shown in the UI. - Extend parse_expr to resolve `datetime.*` attribute access (alongside the existing `wmill.*` handling) - Add Typ::Date variant for `datetime.date` → JSON schema format "date" - Update python worker to import and convert `date.fromisoformat()` - Update argSigToJsonSchemaType, AI types, schema validation, and SQL datatype wasm for the new Date variant Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * all * all --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
dc5e69481d |
test: add end-to-end trigger integration tests and DB CRUD tests
Add 7 #[ignore] e2e tests (one per trigger type) that fire real messages to external services and verify job creation in v2_job. Also add 9 DB-level CRUD tests for MQTT, GCP, and Email triggers. Includes helper shell scripts in tests/fixtures/ to start/stop each external service (MQTT, WebSocket, Postgres replication, Kafka, NATS, SQS via LocalStack, GCP Pub/Sub emulator). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
84d40ba8fd |
test: expand workspace endpoint coverage to ~40 endpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
862ccbbac9 |
test: expand endpoint coverage with workspaces, global users, and unauthed routes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
c8f539bb57 |
test: add integration tests for schedules, groups, folders, users, and drafts
Also extends existing tests with additional endpoint coverage: - scripts: archive/h, delete/h - flows: get/v/:version - apps: get/v/:version, custom_path_exists - resources: list_names/:type - base fixture: add password entry for whoami LEFT JOIN Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
c787ac9a9c | add back mcp_tools endpoints | ||
|
Ruben Fiszel
|
6cd1d3e050 |
chore: remove obsolete debouncing tests
The test file used old JobPayload/FlowValue field names that were refactored into DebouncingSettings/ConcurrencySettings structs. Remove the test file, fixture, feature flag, and cfg gate in monitor.rs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
e655a06563 |
fix: prevent schedule pool connection exhaustion (#7821)
* fix: prevent schedule pool connection exhaustion Schedules were disabling themselves after upgrading from v1.605.0 to v1.614.0 due to pool connection deadlock. The root cause was fetch_authed_from_permissioned_as acquiring a pool connection inside push() while a transaction already held one, exhausting the pool under pressure. Fix: pre-compute Authed before db.begin() for the normal path, and reuse the transaction connection via fetch_authed_from_permissioned_as_conn for the on_behalf_of_email path. Peak pool usage drops from 2 to 1 for all schedule push paths. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: replace pool exhaustion tests with comprehensive schedule push tests Replace the 16 pool-specific tests with 19 tests covering all schedule push code paths: script/flow scheduling, on_behalf_of_email (script and flow), retry wrapping, duplicate detection, invalid timezone/cron/args, script/flow not found, paused schedules, clock shift detection, disabled schedule, path mismatch, push failure disabling schedule, and trigger metadata. Also simplify the obo_authed pattern in push_scheduled_job to use a single match assignment instead of two bindings with .or() chaining. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
hugocasa
|
ef89a51f3a |
feat: upgrade bun to v1.3.8 with regression tests (#7761)
* test: add bun executor tests with minimal production code changes - Add comprehensive bun job tests (bun_jobs.rs) covering: - Basic execution, error handling, annotation modes - Relative imports, deeply nested imports - Dedicated worker protocol for both Node.js and Bun runtimes - Builder tests for lockfile generation (import scanning) - Minimize changes to bun_executor.rs by exposing: - RELATIVE_BUN_LOADER and RELATIVE_BUN_BUILDER constants - build_loader() function and LoaderMode enum - BUN_DEDICATED_WORKER_ARGS constant - generate_dedicated_worker_wrapper() function - Tests call production code directly (build_loader) instead of duplicating script generation logic Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * nit * fix: reuse BUN_PATH/NODE_BIN_PATH from windmill-worker, add node to CI - Tests now use exported BUN_PATH and NODE_BIN_PATH constants instead of duplicating env var logic - Update backend-test.yml: - Upgrade bun to v1.3.8 - Add setup-node action - Add NODE_BIN_PATH to cargo test command Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * add private repo test * fix private repo test * try fix again * fix --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6ec4b92999 |
test: add integration tests for instance group and workspace auto-add interaction
Add tests verifying the interaction between instance groups and workspace auto-add functionality: - Users in instance groups get auto-added to configured workspaces - Role assignment (admin/operator/developer) works correctly - Role precedence when user belongs to multiple groups - User removal when removed from instance group - Cleanup when instance groups removed from workspace config - added_via field tracking Tests are ignored by default in CI and can be run locally with: cargo test -p windmill --test instance_group_auto_add --features private,enterprise -- --ignored Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
865ab70c89 |
add comprehensive permissions tests and restrict operator script archiving
- Add permissions test suite covering scripts, flows, apps, resources, variables, schedules, and folder-based permissions - Add test for operator restrictions (cannot create/update/archive) - Restrict operators from archiving scripts (archive_script_by_path, archive_script_by_hash) for security consistency with flows/apps Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
1b9d1c56c7 |
feat: add HashiCorp Vault secret storage integration (#7599)
* feat: add HashiCorp Vault secret storage integration - Create SecretBackend trait abstraction for secret storage - Add VaultBackend implementation with CRUD operations - Integrate secret backend into variable CRUD operations - Add migration functions (DB → Vault and Vault → DB) - Add frontend configuration UI for secret backend - Add test connection and migration endpoints |
||
|
Pyra
|
d38c96db36 |
feat: workspace dependencies (#7124)
* commit raw requirements Signed-off-by: pyranota <pyra@duck.com> * raw requirements Signed-off-by: pyranota <pyra@duck.com> * implement `parse_annotation` Signed-off-by: pyranota <pyra@duck.com> * more progress on wdeps Signed-off-by: pyranota <pyra@duck.com> * more progress Signed-off-by: pyranota <pyra@duck.com> * fixes Signed-off-by: pyranota <pyra@duck.com> * more progress Signed-off-by: pyranota <pyra@duck.com> * fixes Signed-off-by: pyranota <pyra@duck.com> * cli improvements + raw deps Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fix python versions Signed-off-by: pyranota <pyra@duck.com> * progress Signed-off-by: pyranota <pyra@duck.com> * update :) Signed-off-by: pyranota <pyra@duck.com> * add MaybeLock Signed-off-by: pyranota <pyra@duck.com> * go WIP Signed-off-by: pyranota <pyra@duck.com> * fix python ignoring py version from requirements Signed-off-by: pyranota <pyra@duck.com> * optimize php Signed-off-by: pyranota <pyra@duck.com> * require admin to alter Signed-off-by: pyranota <pyra@duck.com> * fix(cli): flow generateLocks raw deps Signed-off-by: pyranota <pyra@duck.com> * progress in checklist Signed-off-by: pyranota <pyra@duck.com> * fix agent workers Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * nit: remove default features Signed-off-by: pyranota <pyra@duck.com> * oh-wow Signed-off-by: pyranota <pyra@duck.com> * remove dbg! Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * add indexes Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * remove todos Signed-off-by: pyranota <pyra@duck.com> * fix cli Signed-off-by: pyranota <pyra@duck.com> * add debug flag Signed-off-by: pyranota <pyra@duck.com> * cli: remove noise Signed-off-by: pyranota <pyra@duck.com> * fix cli Signed-off-by: pyranota <pyra@duck.com> * remove todos Signed-off-by: pyranota <pyra@duck.com> * trigger deps correctly Signed-off-by: pyranota <pyra@duck.com> * fix frontend Signed-off-by: pyranota <pyra@duck.com> * fix frontend again Signed-off-by: pyranota <pyra@duck.com> * finally fix frontend Signed-off-by: pyranota <pyra@duck.com> * ee repo ref Signed-off-by: pyranota <pyra@duck.com> * fix all Signed-off-by: pyranota <pyra@duck.com> * more fixes... Signed-off-by: pyranota <pyra@duck.com> * remove test Signed-off-by: pyranota <pyra@duck.com> * Update backend-test.yml * comment out legacy test Signed-off-by: pyranota <pyra@duck.com> * fix ci Signed-off-by: pyranota <pyra@duck.com> * fix ci? Signed-off-by: pyranota <pyra@duck.com> * comment out thing Signed-off-by: pyranota <pyra@duck.com> * ignore test Signed-off-by: pyranota <pyra@duck.com> * ci Signed-off-by: pyranota <pyra@duck.com> * base fixture Signed-off-by: pyranota <pyra@duck.com> * fix regression Signed-off-by: pyranota <pyra@duck.com> * fix docs links Signed-off-by: pyranota <pyra@duck.com> * update min version Signed-off-by: pyranota <pyra@duck.com> * simplify * implement cache for get_latest Signed-off-by: pyranota <pyra@duck.com> * move to workspace settings Signed-off-by: pyranota <pyra@duck.com> * sqlx + migration Signed-off-by: pyranota <pyra@duck.com> * more migrations Signed-off-by: pyranota <pyra@duck.com> * use box pin Signed-off-by: pyranota <pyra@duck.com> * nit Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
hugocasa
|
cf86881528 |
fix(backend): add locks to inline preprocessor/failure/tool modules in flows (#6825)
* fix(backend): add locks to inline preprocessor/failure/tool modules in flows * fix merge * fix tests * fix tests * add test for ai agent tool * nit * fix build * fix locking |
||
|
Pyra
|
edece035f8 |
feat: job debouncing (#6878)
* feat(EE): job debouncing Signed-off-by: pyranota <pyra@duck.com> * remove 'script' file Signed-off-by: pyranota <pyra@duck.com> * more work Signed-off-by: pyranota <pyra@duck.com> * properly gate it behind enterprise Signed-off-by: pyranota <pyra@duck.com> * update ee repo ref Signed-off-by: pyranota <pyra@duck.com> * change ee repo ref again Signed-off-by: pyranota <pyra@duck.com> * remove unused variable Signed-off-by: pyranota <pyra@duck.com> * feat(EE): implement TODOs and enhance tracing for job debouncing - Add database index on script(workspace_id, debounce_key) for efficient lookups - Update minimum version requirement to 1.564.0 throughout codebase - Add tracing warnings when debouncing is disabled due to worker version mismatch - Fix all documentation links from TODO placeholders to proper URLs - Replace Gauge icon with Timer icon for debouncing UI elements - Update placeholder text and tooltips with clear descriptions Co-authored-by: Pyra <pyranota@users.noreply.github.com> * create -> crate Signed-off-by: pyranota <pyra@duck.com> * remove index Signed-off-by: pyranota <pyra@duck.com> * some updates Signed-off-by: pyranota <pyra@duck.com> * fix once more Signed-off-by: pyranota <pyra@duck.com> * fix it once more Signed-off-by: pyranota <pyra@duck.com> * Remove flow step debouncing, keep top-level flow debouncing - Remove debounce fields from RawScript and FlowScript FlowModuleValue variants - Remove debounce fields from JobPayload::FlowScript and RawCode - Update raw_script_to_payload function signature - Remove debouncing UI from flow step runtime settings - Remove debouncing toggle handler and indicator badge - Preserve top-level flow debouncing in FlowSettings Co-authored-by: Pyra <pyranota@users.noreply.github.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fixup claude's work Signed-off-by: pyranota <pyra@duck.com> * cleanup: remove dbg! statements, update min version to 1.566.0, add comprehensive comments - Removed all dbg! macro calls from production code - Updated MIN_VERSION_SUPPORTS_DEBOUNCING from 1.564.0 to 1.566.0 - Added comprehensive documentation comments explaining: - Debouncing feature purpose and mechanics - Database schema for debounce_key and debounce_stale_data tables - Version check logic and guard functions - Improved code clarity and maintainability Co-authored-by: Pyra <pyranota@users.noreply.github.com> * improve fallback Signed-off-by: pyranota <pyra@duck.com> * remove comments from old migration Signed-off-by: pyranota <pyra@duck.com> * fix pull Signed-off-by: pyranota <pyra@duck.com> * fix once more Signed-off-by: pyranota <pyra@duck.com> * Update frontend/src/lib/components/ScriptBuilder.svelte Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * add sqlx cache Signed-off-by: pyranota <pyra@duck.com> * fix: frontend CI - fix debounce field names and remove leftover flow step debouncing - Fix ScriptBuilder.svelte: change custom_debounce_key to debounce_key - Add debounce_key and debounce_delay_s fields to NewScript schema in openapi.yaml - Regenerate frontend types from OpenAPI spec - Remove leftover flow step debouncing code from FlowModuleComponent.svelte - Remove debounce fields from RawScript in openflow.openapi.yaml - Remove unused Timer import from FlowModuleHeader.svelte All frontend checks now passing (0 errors, 0 warnings) Co-authored-by: Pyra <pyranota@users.noreply.github.com> * fix ci Signed-off-by: pyranota <pyra@duck.com> * remove unused import Signed-off-by: pyranota <pyra@duck.com> * fix ci again Signed-off-by: pyranota <pyra@duck.com> * udpate ee repo ref Signed-off-by: pyranota <pyra@duck.com> * CI doesn't want to be fixed but I still try Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * ci... Signed-off-by: pyranota <pyra@duck.com> * Update ee-repo-ref.txt * safer migration Signed-off-by: pyranota <pyra@duck.com> * reduce noise in logs Signed-off-by: pyranota <pyra@duck.com> * fix cli for scripts Signed-off-by: pyranota <pyra@duck.com> * nit Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Pyra <pyranota@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Pyra
|
defb6c9694 |
feat: dependency job debouncing (#6769)
* v0 Signed-off-by: pyranota <pyra@duck.com> * optimize relocks * make it work with relative relative imports Signed-off-by: pyranota <pyra@duck.com> * use fallback Signed-off-by: pyranota <pyra@duck.com> * remove dbg and todos Signed-off-by: pyranota <pyra@duck.com> * future proof a bit Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * more cleanup Signed-off-by: pyranota <pyra@duck.com> * remove final TODO Signed-off-by: pyranota <pyra@duck.com> * do not use bytemuck Signed-off-by: pyranota <pyra@duck.com> * optimize hashing Signed-off-by: pyranota <pyra@duck.com> * implementation 1 Signed-off-by: pyranota <pyra@duck.com> * almost v0 Signed-off-by: pyranota <pyra@duck.com> * v0 Signed-off-by: pyranota <pyra@duck.com> * add comments and use fallback Signed-off-by: pyranota <pyra@duck.com> * call dissolve for apps Signed-off-by: pyranota <pyra@duck.com> * add comms Signed-off-by: pyranota <pyra@duck.com> * refactor v0 (partially tested + dirty) Signed-off-by: pyranota <pyra@duck.com> * finishing Signed-off-by: pyranota <pyra@duck.com> * remove TODO Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * silence unused argument Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * implement rebuild_map endpoint Signed-off-by: pyranota <pyra@duck.com> * update windmill api client Signed-off-by: pyranota <pyra@duck.com> * almost finish with tests Signed-off-by: pyranota <pyra@duck.com> * add proper testing Signed-off-by: pyranota <pyra@duck.com> * remove unused fixtures Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * partial cleanup Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-worker/src/scoped_dependency_map.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-common/src/scripts.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * evil doings Signed-off-by: pyranota <pyra@duck.com> * more cleanup * Update SQLx metadata * more cleanup Signed-off-by: pyranota <pyra@duck.com> * fixing CI Signed-off-by: pyranota <pyra@duck.com> * remove python from default features Signed-off-by: pyranota <pyra@duck.com> * feat: dependency job debouncing * checkpoint Signed-off-by: pyranota <pyra@duck.com> * more improvements Signed-off-by: pyranota <pyra@duck.com> * refactor: clean up dependency job debouncing implementation - Add comprehensive comments explaining the debouncing mechanism - Replace debug statements (dbg!) with proper tracing calls - Extract helper functions to reduce code duplication: - extract_to_relock_from_args() for extracting nodes/components - accumulate_debounce_stale_data() for updating stale data - Improve code readability and maintainability Co-authored-by: Pyra <pyranota@users.noreply.github.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * Update backend/windmill-common/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-common/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-queue/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * better error handling for helper Signed-off-by: pyranota <pyra@duck.com> * update ee-ref Signed-off-by: pyranota <pyra@duck.com> * test: add race condition test for dependency job debouncing - Implement test_2 for scripts to test the race condition edge case - Add comprehensive documentation comments to all test functions - Remove empty test_2 stubs for flows (not needed) - Keep test_2 stub for apps with TODO comment The race condition test simulates the scenario where a job is marked as running but debounce_key hasn't been cleaned up yet, forcing the system to create a new job while reusing the existing debounce_key. This edge case can occur due to the lack of transactions in the pull function for performance reasons (see jobs.rs:4415-4425). Co-authored-by: Pyra <pyranota@users.noreply.github.com> * test: implement race condition test for dependency job debouncing - Add comprehensive test_2 for script module that tests the race condition edge case - Remove empty test_2 stubs from flows and apps modules - Fix unused variable warning in worker_lockfiles.rs - Add detailed comments explaining the race condition scenario and test logic Co-authored-by: Pyra <pyranota@users.noreply.github.com> * implement fallback Signed-off-by: pyranota <pyra@duck.com> * make it mostly work * all tests are almost working Signed-off-by: pyranota <pyra@duck.com> * add comments a bit Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-common/src/scripts.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/worker_lockfiles.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/worker_lockfiles.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * feat: improve debouncing documentation, tracing, and error handling - Add comprehensive 3-phase debouncing documentation explaining push/accumulation/pull - Enhance all tracing with structured logging (job_id, workspace_id, node_count, etc.) - Add proper error handling with .map_err() and contextual messages - Replace dbg!() with proper tracing::debug!() - Replace todo!() with proper error handling - Fix typos: 'and edge case' → 'an edge case', 'bc' → 'because' - Fix debug variable name: 'debounce_job_id_0' → 'debounce_job_id_o' - Add documentation for debounce cleanup and stale data retrieval - Add trace-level logging for non-error paths to reduce noise Co-authored-by: Pyra <pyranota@users.noreply.github.com> * do some work for future improvements Signed-off-by: pyranota <pyra@duck.com> * fix tests Signed-off-by: pyranota <pyra@duck.com> * clippy Signed-off-by: pyranota <pyra@duck.com> * update sqlx Signed-off-by: pyranota <pyra@duck.com> * clippy Signed-off-by: pyranota <pyra@duck.com> * update ee ref Signed-off-by: pyranota <pyra@duck.com> * flag tests behind the feature, add timeout Signed-off-by: pyranota <pyra@duck.com> * fix timeout + cleanup Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * row lock debounce_key Signed-off-by: pyranota <pyra@duck.com> * addressing TODOs Signed-off-by: pyranota <pyra@duck.com> * fix test Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * ee repo Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: Pyra <92104930+pyranye@users.noreply.github.com> Co-authored-by: GitHub Action <action@github.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Pyra <pyranota@users.noreply.github.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Pyra
|
ed806bf9d0 |
fix(backend): rework dependency_map handling (#6598)
* v0 Signed-off-by: pyranota <pyra@duck.com> * optimize relocks * make it work with relative relative imports Signed-off-by: pyranota <pyra@duck.com> * use fallback Signed-off-by: pyranota <pyra@duck.com> * remove dbg and todos Signed-off-by: pyranota <pyra@duck.com> * future proof a bit Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * more cleanup Signed-off-by: pyranota <pyra@duck.com> * remove final TODO Signed-off-by: pyranota <pyra@duck.com> * do not use bytemuck Signed-off-by: pyranota <pyra@duck.com> * optimize hashing Signed-off-by: pyranota <pyra@duck.com> * implementation 1 Signed-off-by: pyranota <pyra@duck.com> * almost v0 Signed-off-by: pyranota <pyra@duck.com> * v0 Signed-off-by: pyranota <pyra@duck.com> * add comments and use fallback Signed-off-by: pyranota <pyra@duck.com> * call dissolve for apps Signed-off-by: pyranota <pyra@duck.com> * add comms Signed-off-by: pyranota <pyra@duck.com> * refactor v0 (partially tested + dirty) Signed-off-by: pyranota <pyra@duck.com> * finishing Signed-off-by: pyranota <pyra@duck.com> * remove TODO Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * silence unused argument Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * implement rebuild_map endpoint Signed-off-by: pyranota <pyra@duck.com> * update windmill api client Signed-off-by: pyranota <pyra@duck.com> * almost finish with tests Signed-off-by: pyranota <pyra@duck.com> * add proper testing Signed-off-by: pyranota <pyra@duck.com> * remove unused fixtures Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * partial cleanup Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-worker/src/scoped_dependency_map.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-common/src/scripts.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * evil doings Signed-off-by: pyranota <pyra@duck.com> * more cleanup * Update SQLx metadata * more cleanup Signed-off-by: pyranota <pyra@duck.com> * fixing CI Signed-off-by: pyranota <pyra@duck.com> * remove python from default features Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: Pyra <92104930+pyranye@users.noreply.github.com> Co-authored-by: GitHub Action <action@github.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> |
||
|
pyranota
|
26222539e6 |
feat(python): inline script metadata (PEP 723) (#5712)
* make resolver * more updates * fix build * fix raw_dependencies job type * compat with http agent workers * refactor * rename * more refactor * cleanup * more tests * fix s3 * small fixes * more fixing * fix endpoint * nit: update comment * update ee ref * update ee ref * update ee ref * implement safer `list_available_python_versions` * add tracing to get of authed client * internal: Trigger claude when commenting with /aider (#5783) * add claude instructions files * call claude too when using aider * fix * add draft for linear claude integration * fix build * update ee ref * ignore versions <=3.9 * fix windows build * correct versions filter * fix windows build (this time for real) * inject error to debug CI * update CI * undo debug of CI * fix tests * remove outdated comment * update ee repo ref * Update ee-repo-ref.txt * Update backend/parsers/windmill-parser-py-imports/src/lib.rs Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com> * Update InstanceSetting.svelte --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: centdix <40307056+centdix@users.noreply.github.com> Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com> |
||
|
pyranota
|
0b6d017fed |
feat(python): per import requirement pin (#5520)
* implement single line pin * make panic-safe * use pin even if multiple modules imported withing single statement * add repins and make imports respect pins * keep all pins * Allow multiple pins * add comments + handle stuff more safely * fix fully qualified imports * remove ignore * sort nested * apply unique to output requirements list * fix typo * remove mut * update sqlx * sort imports * sort imports * fix formatter and format * refactor * fix comptime error * write tests * perf: do not capture if string is empty |
||
|
Lucas Abel
|
b1f358d4a2 |
backend: finalize v2 migration (v2 phase 4 - final) (#5155)
* backend: finalize v2 migration (v2 phase 4 - final) * update migration --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Lucas Abel
|
4849af04de |
backend: fix workflow as code (#5239)
* backend: improve `/get_job_update` after v2 * backend: insert missing `workflow_as_code_status` on completion also insert `flow_status` from so we can remove the query when `_duration` is above 500 * backend: fix workflow_as_code after v2 * backend: add `workflow_as_code` worker test |
||
|
Lucas Abel
|
d51c3080f7 | backend: use v2 tables through views where possible (v2 phase 3) (#5119) | ||
|
Lucas Abel
|
3e8e0c653e | backend: migrate job tables to v2 schema (v2 phase 1) (#5084) | ||
|
Lucas Abel
|
977ac5c3f3 |
feat(cache): remove persistent raw values from queue (#4866)
* feat(cache): remove persistent raw values from queue * test: add tests and fix discovered regressions * fix csharp executor after rebase * invalidate script cache after locking * fix lite flow version race * fix flow dependencies lock race * update version flag to `v.1.440` |
||
|
HugoCasa
|
6924370f11 | refactor sql result formatting (#4760) | ||
|
HugoCasa
|
7f2289d4d5 |
feat: flow versioning (#4009)
* feat: flow versioning * fix: sqlx * fix: update schedule test for flow versioning * fix: with_deployment_msg + UI nits * fix: nit * fix: improve down migration * patch: keep latest flow version in flow table for backward compat * fix: app deployments in list view * chore: update ee ref * fix: merge * fix: tests |
||
|
HugoCasa
|
35166883e8 |
feat: bun absolute/relative imports + tests (#2286)
* feat: bun relative imports + tests * fix: add missing .sqlx |
||
|
Ruben Fiszel
|
e051451cd6 | fix: canceling jobs | ||
|
HugoCasa
|
cbbfee8439 |
feat: schedule recovery handler (#2126)
* feat: schedule recovery handler * fix: migration down * fix: rename * fix: adjust + add tests * fix: merge + sqlx prepare * fix: more complete recovery handler * feat: schedule error and recovery times * fix: schedule worker test * feat: slack schedule handlers * fix: update schedule handlers |
||
|
Ruben Fiszel
|
a8c098736b | fix fixtures | ||
|
Kai Jellinghaus
|
d33d465d3a |
fix(backend): bash flow lock & add flow lock tests (#933)
* Fix Bash flow lock * Tests w/ fixes * Add Sequence privileges |
||
|
Kai Jellinghaus
|
6263e90735 | restructure the entire backend layout using workspaces (#815) |