wmill-script-python
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Pyra
|
defb6c9694 |
feat: dependency job debouncing (#6769)
* v0 Signed-off-by: pyranota <pyra@duck.com> * optimize relocks * make it work with relative relative imports Signed-off-by: pyranota <pyra@duck.com> * use fallback Signed-off-by: pyranota <pyra@duck.com> * remove dbg and todos Signed-off-by: pyranota <pyra@duck.com> * future proof a bit Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * more cleanup Signed-off-by: pyranota <pyra@duck.com> * remove final TODO Signed-off-by: pyranota <pyra@duck.com> * do not use bytemuck Signed-off-by: pyranota <pyra@duck.com> * optimize hashing Signed-off-by: pyranota <pyra@duck.com> * implementation 1 Signed-off-by: pyranota <pyra@duck.com> * almost v0 Signed-off-by: pyranota <pyra@duck.com> * v0 Signed-off-by: pyranota <pyra@duck.com> * add comments and use fallback Signed-off-by: pyranota <pyra@duck.com> * call dissolve for apps Signed-off-by: pyranota <pyra@duck.com> * add comms Signed-off-by: pyranota <pyra@duck.com> * refactor v0 (partially tested + dirty) Signed-off-by: pyranota <pyra@duck.com> * finishing Signed-off-by: pyranota <pyra@duck.com> * remove TODO Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * silence unused argument Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * implement rebuild_map endpoint Signed-off-by: pyranota <pyra@duck.com> * update windmill api client Signed-off-by: pyranota <pyra@duck.com> * almost finish with tests Signed-off-by: pyranota <pyra@duck.com> * add proper testing Signed-off-by: pyranota <pyra@duck.com> * remove unused fixtures Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * partial cleanup Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-worker/src/scoped_dependency_map.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-common/src/scripts.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * evil doings Signed-off-by: pyranota <pyra@duck.com> * more cleanup * Update SQLx metadata * more cleanup Signed-off-by: pyranota <pyra@duck.com> * fixing CI Signed-off-by: pyranota <pyra@duck.com> * remove python from default features Signed-off-by: pyranota <pyra@duck.com> * feat: dependency job debouncing * checkpoint Signed-off-by: pyranota <pyra@duck.com> * more improvements Signed-off-by: pyranota <pyra@duck.com> * refactor: clean up dependency job debouncing implementation - Add comprehensive comments explaining the debouncing mechanism - Replace debug statements (dbg!) with proper tracing calls - Extract helper functions to reduce code duplication: - extract_to_relock_from_args() for extracting nodes/components - accumulate_debounce_stale_data() for updating stale data - Improve code readability and maintainability Co-authored-by: Pyra <pyranota@users.noreply.github.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * Update SQLx metadata * Update backend/windmill-common/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-common/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-queue/src/jobs.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * better error handling for helper Signed-off-by: pyranota <pyra@duck.com> * update ee-ref Signed-off-by: pyranota <pyra@duck.com> * test: add race condition test for dependency job debouncing - Implement test_2 for scripts to test the race condition edge case - Add comprehensive documentation comments to all test functions - Remove empty test_2 stubs for flows (not needed) - Keep test_2 stub for apps with TODO comment The race condition test simulates the scenario where a job is marked as running but debounce_key hasn't been cleaned up yet, forcing the system to create a new job while reusing the existing debounce_key. This edge case can occur due to the lack of transactions in the pull function for performance reasons (see jobs.rs:4415-4425). Co-authored-by: Pyra <pyranota@users.noreply.github.com> * test: implement race condition test for dependency job debouncing - Add comprehensive test_2 for script module that tests the race condition edge case - Remove empty test_2 stubs from flows and apps modules - Fix unused variable warning in worker_lockfiles.rs - Add detailed comments explaining the race condition scenario and test logic Co-authored-by: Pyra <pyranota@users.noreply.github.com> * implement fallback Signed-off-by: pyranota <pyra@duck.com> * make it mostly work * all tests are almost working Signed-off-by: pyranota <pyra@duck.com> * add comments a bit Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-common/src/scripts.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/worker_lockfiles.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/worker_lockfiles.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * feat: improve debouncing documentation, tracing, and error handling - Add comprehensive 3-phase debouncing documentation explaining push/accumulation/pull - Enhance all tracing with structured logging (job_id, workspace_id, node_count, etc.) - Add proper error handling with .map_err() and contextual messages - Replace dbg!() with proper tracing::debug!() - Replace todo!() with proper error handling - Fix typos: 'and edge case' → 'an edge case', 'bc' → 'because' - Fix debug variable name: 'debounce_job_id_0' → 'debounce_job_id_o' - Add documentation for debounce cleanup and stale data retrieval - Add trace-level logging for non-error paths to reduce noise Co-authored-by: Pyra <pyranota@users.noreply.github.com> * do some work for future improvements Signed-off-by: pyranota <pyra@duck.com> * fix tests Signed-off-by: pyranota <pyra@duck.com> * clippy Signed-off-by: pyranota <pyra@duck.com> * update sqlx Signed-off-by: pyranota <pyra@duck.com> * clippy Signed-off-by: pyranota <pyra@duck.com> * update ee ref Signed-off-by: pyranota <pyra@duck.com> * flag tests behind the feature, add timeout Signed-off-by: pyranota <pyra@duck.com> * fix timeout + cleanup Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * row lock debounce_key Signed-off-by: pyranota <pyra@duck.com> * addressing TODOs Signed-off-by: pyranota <pyra@duck.com> * fix test Signed-off-by: pyranota <pyra@duck.com> * ee ref Signed-off-by: pyranota <pyra@duck.com> * ee repo Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: Pyra <92104930+pyranye@users.noreply.github.com> Co-authored-by: GitHub Action <action@github.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Pyra <pyranota@users.noreply.github.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |