wmill-script-python
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
hugocasa
|
dc0e59f432 |
feat: add preprocessor support for dedicated workers and bunnative scripts (#8284)
* feat: add preprocessor support for dedicated workers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update ee-repo-ref.txt Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: extract transform_and_run helper in python dedicated wrapper Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add preprocessor support for bunnative scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: resolve unused postprocessor variable in python wrapper Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: add workflow_dispatch trigger to backend integration tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: bunnative fixture lock format and PrewarmedIsolate::spawn callers Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: update generate_dedicated_worker_wrapper callers in bun_jobs test Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: use non-dedicated workers in preprocessor integration tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: prewarm preprocessor isolate for bunnative dedicated workers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: flatten bunnative dedicated worker preprocessing into single result path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use labeled block instead of async block for EE compatibility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update commit reference in ee-repo-ref.txt * chore: update ee-repo-ref to e36945b987f7904fa984181baf3124e7b2722bd1 This commit updates the EE repository reference after PR #445 was merged in windmill-ee-private. Previous ee-repo-ref: 8a2625833452aadb8907242bf502b24ca2dffd73 New ee-repo-ref: e36945b987f7904fa984181baf3124e7b2722bd1 Automated by sync-ee-ref workflow. * Fix merge conflict in ee-repo-ref.txt Resolve merge conflict in ee-repo-ref.txt --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
ea52a8b8ce |
refactor: move integration tests to subcrates to reduce recompilation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Pyra
|
7249b82dba | feat: better mixed versions handling (#7628) | ||
|
Pyra
|
9d698dabb4 |
feat: v2 job debouncing (#7411)
* rework everything again Signed-off-by: pyranota <pyra@duck.com> * updcate sqlx Signed-off-by: pyranota <pyra@duck.com> * update ref Signed-off-by: pyranota <pyra@duck.com> * fix things Signed-off-by: pyranota <pyra@duck.com> * fix function Signed-off-by: pyranota <pyra@duck.com> * final fixes Signed-off-by: pyranota <pyra@duck.com> * update sqlx Signed-off-by: pyranota <pyra@duck.com> * fix script creation Signed-off-by: pyranota <pyra@duck.com> * address todo Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * remove dbg Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fix Signed-off-by: pyranota <pyra@duck.com> * fixups Signed-off-by: pyranota <pyra@duck.com> * fix cargo.toml Signed-off-by: pyranota <pyra@duck.com> * update ee repo Signed-off-by: pyranota <pyra@duck.com> * fix ci Signed-off-by: pyranota <pyra@duck.com> * nit Signed-off-by: pyranota <pyra@duck.com> * ref Signed-off-by: pyranota <pyra@duck.com> * fix Signed-off-by: pyranota <pyra@duck.com> * ee repo Signed-off-by: pyranota <pyra@duck.com> * sqlx Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * remove dbg Signed-off-by: pyranota <pyra@duck.com> * sqlx Signed-off-by: pyranota <pyra@duck.com> * feat: v2 job debouncing Signed-off-by: pyranota <pyra@duck.com> * v2 debouncing Signed-off-by: pyranota <pyra@duck.com> * compat Signed-off-by: pyranota <pyra@duck.com> * fix all Signed-off-by: pyranota <pyra@duck.com> * remove legacy logic Signed-off-by: pyranota <pyra@duck.com> * nit Signed-off-by: pyranota <pyra@duck.com> * finish Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * stage import Signed-off-by: pyranota <pyra@duck.com> * remove unused import Signed-off-by: pyranota <pyra@duck.com> * remove comments Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-queue/src/jobs.rs Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * Update backend/windmill-common/src/runnable_settings/settings.rs Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * Update frontend/src/lib/components/flows/DebounceLimit.svelte Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * fix nits Signed-off-by: pyranota <pyra@duck.com> * force no compat in ci Signed-off-by: pyranota <pyra@duck.com> * better default key handling Signed-off-by: pyranota <pyra@duck.com> * simplify Signed-off-by: pyranota <pyra@duck.com> * sqlx Signed-off-by: pyranota <pyra@duck.com> * optimize Signed-off-by: pyranota <pyra@duck.com> * sqlx Signed-off-by: pyranota <pyra@duck.com> * fix flow step Signed-off-by: pyranota <pyra@duck.com> * add openapi Signed-off-by: pyranota <pyra@duck.com> * fix ci Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * rename WMDEBUG flag Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * remove gate for is_flow_step on pull Signed-off-by: pyranota <pyra@duck.com> * remove dead code Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * chore: update ee-repo-ref to 71d872d27de44227b82f75a9c0879d8428684880 This commit updates the EE repository reference after PR #369 was merged in windmill-ee-private. Previous ee-repo-ref: 4c7642dc9f7f4467fc055136bd56558b66ef2d54 New ee-repo-ref: 71d872d27de44227b82f75a9c0879d8428684880 Automated by sync-ee-ref workflow. --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasa
|
a6993823af |
feat: restart flow from step with different flow version (#7409)
* feat: restart flow from step with different flow version * fix tests * fix tests |
||
|
Pyra
|
fe56191422 |
feat(internal): runnable settings (#7298)
* rework everything again Signed-off-by: pyranota <pyra@duck.com> * updcate sqlx Signed-off-by: pyranota <pyra@duck.com> * update ref Signed-off-by: pyranota <pyra@duck.com> * fix things Signed-off-by: pyranota <pyra@duck.com> * fix function Signed-off-by: pyranota <pyra@duck.com> * final fixes Signed-off-by: pyranota <pyra@duck.com> * update sqlx Signed-off-by: pyranota <pyra@duck.com> * fix script creation Signed-off-by: pyranota <pyra@duck.com> * address todo Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * remove dbg Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fix Signed-off-by: pyranota <pyra@duck.com> * fixups Signed-off-by: pyranota <pyra@duck.com> * fix cargo.toml Signed-off-by: pyranota <pyra@duck.com> * update ee repo Signed-off-by: pyranota <pyra@duck.com> * fix ci Signed-off-by: pyranota <pyra@duck.com> * nit Signed-off-by: pyranota <pyra@duck.com> * ref Signed-off-by: pyranota <pyra@duck.com> * fix Signed-off-by: pyranota <pyra@duck.com> * ee repo Signed-off-by: pyranota <pyra@duck.com> * sqlx Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * remove dbg Signed-off-by: pyranota <pyra@duck.com> * sqlx Signed-off-by: pyranota <pyra@duck.com> * chore: update ee-repo-ref to 505eadbff32d102ea5245a2bef88ce6f1bb95395 This commit updates the EE repository reference after PR #348 was merged in windmill-ee-private. Previous ee-repo-ref: 195243e56cc0eab55f8890fa57297206bfe2c18c New ee-repo-ref: 505eadbff32d102ea5245a2bef88ce6f1bb95395 Automated by sync-ee-ref workflow. * ci: force runnable settings Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Diego Imbert
|
7b1a8b0dce | fix: improve S3 etag caching (#7301) | ||
|
Pyra
|
1dfb0f0078 |
refactor: move concurrency and debouncing settings to their own structs (#7286)
* move concurrency and debouncing settings to it's own structs Signed-off-by: pyranota <pyra@duck.com> * cleanup queue::jobs::pull Signed-off-by: pyranota <pyra@duck.com> * done Signed-off-by: pyranota <pyra@duck.com> * migrated the rest Signed-off-by: pyranota <pyra@duck.com> * implement todo! * impmenet todo! v2 * more cleanup * fix all Signed-off-by: pyranota <pyra@duck.com> * remove unrelated code Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * do not serialize rawcode Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fixes Signed-off-by: pyranota <pyra@duck.com> * update ee repo ref Signed-off-by: pyranota <pyra@duck.com> * update ee repo Signed-off-by: pyranota <pyra@duck.com> * fix tests Signed-off-by: pyranota <pyra@duck.com> * Update ee-repo-ref.txt --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Pyra
|
edece035f8 |
feat: job debouncing (#6878)
* feat(EE): job debouncing Signed-off-by: pyranota <pyra@duck.com> * remove 'script' file Signed-off-by: pyranota <pyra@duck.com> * more work Signed-off-by: pyranota <pyra@duck.com> * properly gate it behind enterprise Signed-off-by: pyranota <pyra@duck.com> * update ee repo ref Signed-off-by: pyranota <pyra@duck.com> * change ee repo ref again Signed-off-by: pyranota <pyra@duck.com> * remove unused variable Signed-off-by: pyranota <pyra@duck.com> * feat(EE): implement TODOs and enhance tracing for job debouncing - Add database index on script(workspace_id, debounce_key) for efficient lookups - Update minimum version requirement to 1.564.0 throughout codebase - Add tracing warnings when debouncing is disabled due to worker version mismatch - Fix all documentation links from TODO placeholders to proper URLs - Replace Gauge icon with Timer icon for debouncing UI elements - Update placeholder text and tooltips with clear descriptions Co-authored-by: Pyra <pyranota@users.noreply.github.com> * create -> crate Signed-off-by: pyranota <pyra@duck.com> * remove index Signed-off-by: pyranota <pyra@duck.com> * some updates Signed-off-by: pyranota <pyra@duck.com> * fix once more Signed-off-by: pyranota <pyra@duck.com> * fix it once more Signed-off-by: pyranota <pyra@duck.com> * Remove flow step debouncing, keep top-level flow debouncing - Remove debounce fields from RawScript and FlowScript FlowModuleValue variants - Remove debounce fields from JobPayload::FlowScript and RawCode - Update raw_script_to_payload function signature - Remove debouncing UI from flow step runtime settings - Remove debouncing toggle handler and indicator badge - Preserve top-level flow debouncing in FlowSettings Co-authored-by: Pyra <pyranota@users.noreply.github.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fixup claude's work Signed-off-by: pyranota <pyra@duck.com> * cleanup: remove dbg! statements, update min version to 1.566.0, add comprehensive comments - Removed all dbg! macro calls from production code - Updated MIN_VERSION_SUPPORTS_DEBOUNCING from 1.564.0 to 1.566.0 - Added comprehensive documentation comments explaining: - Debouncing feature purpose and mechanics - Database schema for debounce_key and debounce_stale_data tables - Version check logic and guard functions - Improved code clarity and maintainability Co-authored-by: Pyra <pyranota@users.noreply.github.com> * improve fallback Signed-off-by: pyranota <pyra@duck.com> * remove comments from old migration Signed-off-by: pyranota <pyra@duck.com> * fix pull Signed-off-by: pyranota <pyra@duck.com> * fix once more Signed-off-by: pyranota <pyra@duck.com> * Update frontend/src/lib/components/ScriptBuilder.svelte Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * add sqlx cache Signed-off-by: pyranota <pyra@duck.com> * fix: frontend CI - fix debounce field names and remove leftover flow step debouncing - Fix ScriptBuilder.svelte: change custom_debounce_key to debounce_key - Add debounce_key and debounce_delay_s fields to NewScript schema in openapi.yaml - Regenerate frontend types from OpenAPI spec - Remove leftover flow step debouncing code from FlowModuleComponent.svelte - Remove debounce fields from RawScript in openflow.openapi.yaml - Remove unused Timer import from FlowModuleHeader.svelte All frontend checks now passing (0 errors, 0 warnings) Co-authored-by: Pyra <pyranota@users.noreply.github.com> * fix ci Signed-off-by: pyranota <pyra@duck.com> * remove unused import Signed-off-by: pyranota <pyra@duck.com> * fix ci again Signed-off-by: pyranota <pyra@duck.com> * udpate ee repo ref Signed-off-by: pyranota <pyra@duck.com> * CI doesn't want to be fixed but I still try Signed-off-by: pyranota <pyra@duck.com> * nits Signed-off-by: pyranota <pyra@duck.com> * ci... Signed-off-by: pyranota <pyra@duck.com> * Update ee-repo-ref.txt * safer migration Signed-off-by: pyranota <pyra@duck.com> * reduce noise in logs Signed-off-by: pyranota <pyra@duck.com> * fix cli for scripts Signed-off-by: pyranota <pyra@duck.com> * nit Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Pyra <pyranota@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Pyra
|
defb6c9694 |
feat: dependency job debouncing (#6769)
* v0 Signed-off-by: pyranota <pyra@duck.com> * optimize relocks * make it work with relative relative imports Signed-off-by: pyranota <pyra@duck.com> * use fallback Signed-off-by: pyranota <pyra@duck.com> * remove dbg and todos Signed-off-by: pyranota <pyra@duck.com> * future proof a bit Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * more cleanup Signed-off-by: pyranota <pyra@duck.com> * remove final TODO Signed-off-by: pyranota <pyra@duck.com> * do not use bytemuck Signed-off-by: pyranota <pyra@duck.com> * optimize hashing Signed-off-by: pyranota <pyra@duck.com> * implementation 1 Signed-off-by: pyranota <pyra@duck.com> * almost v0 Signed-off-by: pyranota <pyra@duck.com> * v0 Signed-off-by: pyranota <pyra@duck.com> * add comments and use fallback Signed-off-by: pyranota <pyra@duck.com> * call dissolve for apps Signed-off-by: pyranota <pyra@duck.com> * add comms Signed-off-by: pyranota <pyra@duck.com> * refactor v0 (partially tested + dirty) Signed-off-by: pyranota <pyra@duck.com> * finishing Signed-off-by: pyranota <pyra@duck.com> * remove TODO Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * silence unused argument Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * implement rebuild_map endpoint Signed-off-by: pyranota <pyra@duck.com> * update windmill api client Signed-off-by: pyranota <pyra@duck.com> * almost finish with tests Signed-off-by: pyranota <pyra@duck.com> * add proper testing Signed-off-by: pyranota <pyra@duck.com> * remove unused fixtures Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * partial cleanup Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-worker/src/scoped_dependency_map.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-common/src/scripts.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * evil doings Signed-off-by: pyranota <pyra@duck.com> * more cleanup * Update SQLx metadata * more cleanup Signed-off-by: pyranota <pyra@duck.com> * fixing CI Signed-off-by: pyranota <pyra@duck.com> * remove python from default features Signed-off-by: pyranota <pyra@duck.com> * feat: dependency job debouncing * checkpoint Signed-off-by: pyranota <pyra@duck.com> * more improvements Signed-off-by: pyranota <pyra@duck.com> * refactor: clean up dependency job debouncing implementation - Add comprehensive comments explaining the debouncing mechanism - Replace debug statements (dbg!) with proper tracing calls - Extract helper functions to reduce code duplication: - extract_to_relock_from_args() for extracting nodes/components - accumulate_debounce_stale_data() for updating stale data - Improve code readability and maintainability Co-authored-by: Pyra <pyranota@users.noreply.github.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * Update backend/windmill-common/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-common/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-queue/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * better error handling for helper Signed-off-by: pyranota <pyra@duck.com> * update ee-ref Signed-off-by: pyranota <pyra@duck.com> * test: add race condition test for dependency job debouncing - Implement test_2 for scripts to test the race condition edge case - Add comprehensive documentation comments to all test functions - Remove empty test_2 stubs for flows (not needed) - Keep test_2 stub for apps with TODO comment The race condition test simulates the scenario where a job is marked as running but debounce_key hasn't been cleaned up yet, forcing the system to create a new job while reusing the existing debounce_key. This edge case can occur due to the lack of transactions in the pull function for performance reasons (see jobs.rs:4415-4425). Co-authored-by: Pyra <pyranota@users.noreply.github.com> * test: implement race condition test for dependency job debouncing - Add comprehensive test_2 for script module that tests the race condition edge case - Remove empty test_2 stubs from flows and apps modules - Fix unused variable warning in worker_lockfiles.rs - Add detailed comments explaining the race condition scenario and test logic Co-authored-by: Pyra <pyranota@users.noreply.github.com> * implement fallback Signed-off-by: pyranota <pyra@duck.com> * make it mostly work * all tests are almost working Signed-off-by: pyranota <pyra@duck.com> * add comments a bit Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-common/src/scripts.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/worker_lockfiles.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/worker_lockfiles.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * feat: improve debouncing documentation, tracing, and error handling - Add comprehensive 3-phase debouncing documentation explaining push/accumulation/pull - Enhance all tracing with structured logging (job_id, workspace_id, node_count, etc.) - Add proper error handling with .map_err() and contextual messages - Replace dbg!() with proper tracing::debug!() - Replace todo!() with proper error handling - Fix typos: 'and edge case' → 'an edge case', 'bc' → 'because' - Fix debug variable name: 'debounce_job_id_0' → 'debounce_job_id_o' - Add documentation for debounce cleanup and stale data retrieval - Add trace-level logging for non-error paths to reduce noise Co-authored-by: Pyra <pyranota@users.noreply.github.com> * do some work for future improvements Signed-off-by: pyranota <pyra@duck.com> * fix tests Signed-off-by: pyranota <pyra@duck.com> * clippy Signed-off-by: pyranota <pyra@duck.com> * update sqlx Signed-off-by: pyranota <pyra@duck.com> * clippy Signed-off-by: pyranota <pyra@duck.com> * update ee ref Signed-off-by: pyranota <pyra@duck.com> * flag tests behind the feature, add timeout Signed-off-by: pyranota <pyra@duck.com> * fix timeout + cleanup Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * row lock debounce_key Signed-off-by: pyranota <pyra@duck.com> * addressing TODOs Signed-off-by: pyranota <pyra@duck.com> * fix test Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * ee repo Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: Pyra <92104930+pyranye@users.noreply.github.com> Co-authored-by: GitHub Action <action@github.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Pyra <pyranota@users.noreply.github.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Ruben Fiszel
|
cf70265ee9 |
rust tests cleanup (#6573)
* test improvements * improve tests * improve tests |