kalugin66/windmill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10,709 Commits 513 Branches 1,362 Tags
dfd08d8a4bcf270a0579e68906f207e15ad52da0
Commit Graph

4910 Commits

Author SHA1 Message Date
Ruben Fiszel
dfd08d8a4b refactor: improve usage table behavior to eliminate synchronous row locks (#6863) 
* refactor: improve usage table behavior to eliminate synchronous row locks

Replace synchronous INSERT...RETURNING with SELECT + async UPDATE pattern:
- Add check_usage_limits() to read current usage without row locks
- Add increment_usage_async() to update usage in background task
- Refactor job push logic to use optimistic validation
- Simplify job completion tracking with better error handling

This eliminates blocking row locks on the usage table during job creation,
significantly improving throughput and reducing contention.

Note: Requires running 'cargo sqlx prepare' with database access to update
the query cache in .sqlx/ directory.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* Update SQLx metadata

* refactor: optimize cloud usage checks with caching and conditional queries

- Add 60s cache for superadmin status checks to reduce DB load
- Skip unnecessary user usage query for premium workspaces
- Use existing team plan status cache (already implemented in windmill-common)
- Update check_usage_limits to accept check_user_usage parameter
- Add sqlx query cache for conditional user usage query

This optimization eliminates redundant database queries during job creation,
particularly for premium workspaces where user usage tracking is not needed.

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2025-10-18 07:47:31 +00:00
Ruben Fiszel
4cc2166c10 chore(main): release 1.563.3 (#6864) 
* chore(main): release 1.563.3

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-18 06:29:11 +00:00
Ruben Fiszel
200242aa3b chore(main): release 1.563.2 (#6862) 
* chore(main): release 1.563.2

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-17 18:30:16 +00:00
Ruben Fiszel
f581ce6248 fix: monitor less frequent cleanup process periodicity 2025-10-17 18:27:11 +00:00
Ruben Fiszel
4fdbd9184e chore(main): release 1.563.1 (#6861) 
* chore(main): release 1.563.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-17 17:21:32 +00:00
Ruben Fiszel
c8d2a036b6 re-route agent workers job if needed prior to applying concurrency limits 2025-10-17 17:15:05 +00:00
Ruben Fiszel
6faea9adad fix: fix concurrency limit behavior with remote agents dep jobs 2025-10-17 16:59:29 +00:00
Ruben Fiszel
c3299e8b8b chore(main): release 1.563.0 (#6856) 
* chore(main): release 1.563.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-17 14:27:49 +00:00
hugocasa
8cd0006498 feat: http routes streaming (#6834) 
* feat: http routes streaming

* improve webhook config section for SSE

* feat: add request type config to http routes

* fix check

* sqlx

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-10-17 14:05:17 +00:00
Ruben Fiszel
d2ae0533f4 fix select grant 2025-10-17 13:59:58 +00:00
wendrul
e719be54e0 Fix ansible delegate mode only working with ssh (#6858) 2025-10-17 13:58:00 +00:00
Ruben Fiszel
60e17e5068 fix: add grant select on debounce_key 2025-10-17 13:55:02 +00:00
Pyra
2868eeda26 perf: remove unnecessary db call (#6853) 
* perf: remove unnecessary db call

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2025-10-17 11:51:55 +00:00
Ruben Fiszel
5c02af11c3 remove benchmark from all_features_oss 2025-10-17 08:28:06 +00:00
Ruben Fiszel
34cd68676b fix migration error 2025-10-17 03:42:41 +00:00
Ruben Fiszel
cc64f8acef chore(main): release 1.562.0 (#6841) 
* chore(main): release 1.562.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-16 21:54:50 +00:00
Diego Imbert
72c6bad0dd create ducklake_user_pg_pwd if deleted by CLI (#6845) 2025-10-16 21:54:00 +00:00
Ruben Fiszel
ea8c9cc766 remove safety migration 2025-10-16 21:29:44 +00:00
hugocasa
c86b3448b8 feat: add support for sage intacct oauth (#6794) 
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-10-16 20:40:25 +00:00
Diego Imbert
a4a502cf3a ducklake safety for instance_settings.yaml users (#6844) 2025-10-16 20:39:11 +00:00
Ruben Fiszel
e5a11a7bc6 split main async with a box pin 2025-10-16 20:31:46 +00:00
Ruben Fiszel
3dd75ad18d fix rare stack overflow bc of async size 2025-10-16 20:05:09 +00:00
Ruben Fiszel
6543a83d9f fix rare stack overflow bc of async size 2025-10-16 20:01:30 +00:00
wendrul
83337b2035 Ansible repo update hubPath + fixes (#6843) 
* Allow ssh auth

* ssh changes pt2

* Fix unused

* Update package.json and hubPaths
 2025-10-16 18:24:41 +00:00
Pyra
952a15a877 remove dbg! leftovers (#6842) 
Signed-off-by: pyranota <pyra@duck.com>
 2025-10-16 17:29:57 +00:00
Ruben Fiszel
0c4e14e81c fix compile 2025-10-16 17:21:09 +00:00
Pyra
defb6c9694 feat: dependency job debouncing (#6769) 
* v0

Signed-off-by: pyranota <pyra@duck.com>

* optimize relocks

* make it work with relative relative imports

Signed-off-by: pyranota <pyra@duck.com>

* use fallback

Signed-off-by: pyranota <pyra@duck.com>

* remove dbg and todos

Signed-off-by: pyranota <pyra@duck.com>

* future proof a bit

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* more cleanup

Signed-off-by: pyranota <pyra@duck.com>

* remove final TODO

Signed-off-by: pyranota <pyra@duck.com>

* do not use bytemuck

Signed-off-by: pyranota <pyra@duck.com>

* optimize hashing

Signed-off-by: pyranota <pyra@duck.com>

* implementation 1

Signed-off-by: pyranota <pyra@duck.com>

* almost v0

Signed-off-by: pyranota <pyra@duck.com>

* v0

Signed-off-by: pyranota <pyra@duck.com>

* add comments and use fallback

Signed-off-by: pyranota <pyra@duck.com>

* call dissolve for apps

Signed-off-by: pyranota <pyra@duck.com>

* add comms

Signed-off-by: pyranota <pyra@duck.com>

* refactor v0 (partially tested + dirty)

Signed-off-by: pyranota <pyra@duck.com>

* finishing

Signed-off-by: pyranota <pyra@duck.com>

* remove TODO

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* silence unused argument

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* implement rebuild_map endpoint

Signed-off-by: pyranota <pyra@duck.com>

* update windmill api client

Signed-off-by: pyranota <pyra@duck.com>

* almost finish with tests

Signed-off-by: pyranota <pyra@duck.com>

* add proper testing

Signed-off-by: pyranota <pyra@duck.com>

* remove unused fixtures

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* partial cleanup

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-worker/src/scoped_dependency_map.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/scripts.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* evil doings

Signed-off-by: pyranota <pyra@duck.com>

* more cleanup

* Update SQLx metadata

* more cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fixing CI

Signed-off-by: pyranota <pyra@duck.com>

* remove python from default features

Signed-off-by: pyranota <pyra@duck.com>

* feat: dependency job debouncing

* checkpoint

Signed-off-by: pyranota <pyra@duck.com>

* more improvements

Signed-off-by: pyranota <pyra@duck.com>

* refactor: clean up dependency job debouncing implementation

- Add comprehensive comments explaining the debouncing mechanism
- Replace debug statements (dbg!) with proper tracing calls
- Extract helper functions to reduce code duplication:
  - extract_to_relock_from_args() for extracting nodes/components
  - accumulate_debounce_stale_data() for updating stale data
- Improve code readability and maintainability

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* Update backend/windmill-common/src/jobs.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/jobs.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-queue/src/jobs.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* better error handling for helper

Signed-off-by: pyranota <pyra@duck.com>

* update ee-ref

Signed-off-by: pyranota <pyra@duck.com>

* test: add race condition test for dependency job debouncing

- Implement test_2 for scripts to test the race condition edge case
- Add comprehensive documentation comments to all test functions
- Remove empty test_2 stubs for flows (not needed)
- Keep test_2 stub for apps with TODO comment

The race condition test simulates the scenario where a job is marked
as running but debounce_key hasn't been cleaned up yet, forcing the
system to create a new job while reusing the existing debounce_key.

This edge case can occur due to the lack of transactions in the pull
function for performance reasons (see jobs.rs:4415-4425).

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* test: implement race condition test for dependency job debouncing

- Add comprehensive test_2 for script module that tests the race condition edge case
- Remove empty test_2 stubs from flows and apps modules
- Fix unused variable warning in worker_lockfiles.rs
- Add detailed comments explaining the race condition scenario and test logic

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* implement fallback

Signed-off-by: pyranota <pyra@duck.com>

* make it mostly work

* all tests are almost working

Signed-off-by: pyranota <pyra@duck.com>

* add comments a bit

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-common/src/scripts.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/worker_lockfiles.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/worker_lockfiles.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* feat: improve debouncing documentation, tracing, and error handling

- Add comprehensive 3-phase debouncing documentation explaining push/accumulation/pull
- Enhance all tracing with structured logging (job_id, workspace_id, node_count, etc.)
- Add proper error handling with .map_err() and contextual messages
- Replace dbg!() with proper tracing::debug!()
- Replace todo!() with proper error handling
- Fix typos: 'and edge case' → 'an edge case', 'bc' → 'because'
- Fix debug variable name: 'debounce_job_id_0' → 'debounce_job_id_o'
- Add documentation for debounce cleanup and stale data retrieval
- Add trace-level logging for non-error paths to reduce noise

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* do some work for future improvements

Signed-off-by: pyranota <pyra@duck.com>

* fix tests

Signed-off-by: pyranota <pyra@duck.com>

* clippy

Signed-off-by: pyranota <pyra@duck.com>

* update sqlx

Signed-off-by: pyranota <pyra@duck.com>

* clippy

Signed-off-by: pyranota <pyra@duck.com>

* update ee ref

Signed-off-by: pyranota <pyra@duck.com>

* flag tests behind the feature, add timeout

Signed-off-by: pyranota <pyra@duck.com>

* fix timeout + cleanup

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* row lock debounce_key

Signed-off-by: pyranota <pyra@duck.com>

* addressing TODOs

Signed-off-by: pyranota <pyra@duck.com>

* fix test

Signed-off-by: pyranota <pyra@duck.com>

* ee ref

Signed-off-by: pyranota <pyra@duck.com>

* ee repo

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Pyra <92104930+pyranye@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Pyra <pyranota@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-10-16 16:49:35 +00:00
Ruben Fiszel
0c1fbc6f0d fix compile 2025-10-16 16:28:04 +00:00
Ruben Fiszel
d16cc56f58 nit compile 2025-10-16 16:24:36 +00:00
Ruben Fiszel
f723a1fb72 fix: add configurable timeout sse stream 2025-10-16 15:29:49 +00:00
Ruben Fiszel
0fe81f5b98 chore(main): release 1.561.0 (#6838) 
* chore(main): release 1.561.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-16 15:07:59 +00:00
hugocasa
3b5c962473 fix(backend): revert flow node opti for ai agents (#6840) 
* Revert "feat(backend): use flow nodes opti for ai agent steps (#6808)"

This reverts commit 8d5acda340.

* fix(backend): revert flow node opti for ai agents

* keep standard base64
 2025-10-16 14:05:06 +00:00
Ruben Fiszel
aeb6829011 use excluded where relevant 2025-10-16 09:19:44 +00:00
wendrul
32fae7a10c feat: ansible playbook execution git repo mode (repo viewer + UI utils) (#6831) 
* Improve minio on flake.nix

* Add first asset parsing logic for ansible

* Correct html gt sign

* Decouple s3 file picker from drawer

* Factor duplicate code into snippet

* Update S3FilePickerInner to be compatible

* Fix pane shrinking issue

* Git repo viewer

* Change GitRepoViewer

* Endpoints for git repo visualizer

* Move git repo viewer to its own component

* Add button to populate git repo viewer

* Update parser yaml for new ansisble features (repo viewer)

* Reflect parser changes for ansible

* Add button to add the git repo mode of declaration for ansible

* Factor function

* Playbook + inventories into the drawer

* Add button to add inventories from s3

* Move tests to lib.rs

* Inventory loading from s3

* Move get github app token logic to be reused by ansible

* Update parser and ansible executor

* Use the correct path for inventories

* Add nushell to flake for wasm builds

* Add published parser

* Update hubPaths with clone and upload to s3

* Update ee-repo to the branch ref

* Fix npm run check

* Update cargo.lock

* Change labels on buttons

* Remove debug log

* Update ee-repo-ref

* Fix ee issues

* Update ee-repo ref

* Fix typo

* Fix ee

* Update ee-repo-ref

* Fix missing imports

* Unused var

* Fix typo

* Layout improvents

* Fix typos

* Remove unused function and log

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-10-16 09:08:18 +00:00
Ruben Fiszel
06c05200cd chore(main): release 1.560.0 (#6823) 
* chore(main): release 1.560.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-15 22:39:10 +00:00
Alexander Petric
48acc57823 fix: always create instance groups with uuid (#6826) 
* fix: always create instance groups with uuid

* Update SQLx metadata

* repo ref

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2025-10-15 22:30:09 +00:00
Ruben Fiszel
892ce64ea8 fix: fix concurrency key filter 2025-10-15 22:24:11 +00:00
Ruben Fiszel
3d5631938f error 2025-10-15 22:12:09 +00:00
hugocasa
8d5acda340 feat(backend): use flow nodes opti for ai agent steps (#6808) 2025-10-14 18:06:27 +00:00
hugocasa
9d9c29fdfa feat: add support for zoho oauth (#6809) 2025-10-14 17:32:24 +00:00
dieriba
256522273e fix: bug for loop flow inconsistent state (#6815) 
* fix

* Update frontend/src/lib/components/flows/content/FlowLoop.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
 2025-10-14 16:00:15 +00:00
Ruben Fiszel
2736586298 chore(main): release 1.559.0 (#6796) 
* chore(main): release 1.559.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-14 15:27:13 +00:00
centdix
784f988b07 chore(mcp): display API endpoint tools in MCP URL generation (#6811) 
* chore(mcp): display API endpoint tools in MCP URL generation

- Add list of 24 API endpoint tools that are always available via MCP
- Display API endpoints as green badges below scripts/flows list
- Update tooltip to mention 'scripts, flows, and API endpoints'
- Show count of available API endpoints in the UI

API endpoints include operations for variables, resources, scripts, flows,
jobs, schedules, and workers.

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* better

* nit

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2025-10-14 09:22:58 +00:00
Pyra
e2feba391c feat(rust): add resource types (#5843) 
* feat(rust): add resource types

* write tests

* copy from go-resources

* implement frontend for rust

* update

* update cli

* update tests

* update

Signed-off-by: pyranota <pyra@duck.com>

* fix package-lock

* fix ci

* update parser

Signed-off-by: pyranota <pyra@duck.com>

* Update frontend/src/lib/components/Editor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
 2025-10-13 19:15:44 +00:00
wendrul
9149faf305 fix: Safeguard prevents button from deleting non-fork workspaces (#6795) 
* Fix: Safeguard to prevent `delete fork` button from deleting non-fork workspaces

* fix typo

* Consistent prefix
 2025-10-10 18:53:30 +00:00
Alexander Petric
e0d9017036 fix: show workspace prefix to non superadmins for app deploy custom path (#6793) 2025-10-10 18:52:56 +00:00
hugocasa
b972eb9721 feat: add support for contextual vars in SQL (#6791) 2025-10-10 18:52:24 +00:00
Ruben Fiszel
7ebd1bc56c chore(main): release 1.558.1 (#6789) 
* chore(main): release 1.558.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
 2025-10-09 21:08:10 +00:00
Alexander Petric
1c9faf9d03 fix: support pg jsonb array (#6788) 2025-10-09 21:01:43 +00:00
Ruben Fiszel
1faba5b097 make js expr more resilient in case where previous result is error 2025-10-09 20:59:00 +00:00