kalugin66/windmill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10,788 Commits 513 Branches 1,362 Tags
1d3d6acfb34f76efda1d423fb4151bcdd6830471
Commit Graph

646 Commits

Author SHA1 Message Date
Pyra
edece035f8 feat: job debouncing (#6878) 
* feat(EE): job debouncing

Signed-off-by: pyranota <pyra@duck.com>

* remove 'script' file

Signed-off-by: pyranota <pyra@duck.com>

* more work

Signed-off-by: pyranota <pyra@duck.com>

* properly gate it behind enterprise

Signed-off-by: pyranota <pyra@duck.com>

* update ee repo ref

Signed-off-by: pyranota <pyra@duck.com>

* change ee repo ref again

Signed-off-by: pyranota <pyra@duck.com>

* remove unused variable

Signed-off-by: pyranota <pyra@duck.com>

* feat(EE): implement TODOs and enhance tracing for job debouncing

- Add database index on script(workspace_id, debounce_key) for efficient lookups
- Update minimum version requirement to 1.564.0 throughout codebase
- Add tracing warnings when debouncing is disabled due to worker version mismatch
- Fix all documentation links from TODO placeholders to proper URLs
- Replace Gauge icon with Timer icon for debouncing UI elements
- Update placeholder text and tooltips with clear descriptions

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* create -> crate

Signed-off-by: pyranota <pyra@duck.com>

* remove index

Signed-off-by: pyranota <pyra@duck.com>

* some updates

Signed-off-by: pyranota <pyra@duck.com>

* fix once more

Signed-off-by: pyranota <pyra@duck.com>

* fix it once more

Signed-off-by: pyranota <pyra@duck.com>

* Remove flow step debouncing, keep top-level flow debouncing

- Remove debounce fields from RawScript and FlowScript FlowModuleValue variants
- Remove debounce fields from JobPayload::FlowScript and RawCode
- Update raw_script_to_payload function signature
- Remove debouncing UI from flow step runtime settings
- Remove debouncing toggle handler and indicator badge
- Preserve top-level flow debouncing in FlowSettings

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fixup claude's work

Signed-off-by: pyranota <pyra@duck.com>

* cleanup: remove dbg! statements, update min version to 1.566.0, add comprehensive comments

- Removed all dbg! macro calls from production code
- Updated MIN_VERSION_SUPPORTS_DEBOUNCING from 1.564.0 to 1.566.0
- Added comprehensive documentation comments explaining:
  - Debouncing feature purpose and mechanics
  - Database schema for debounce_key and debounce_stale_data tables
  - Version check logic and guard functions
- Improved code clarity and maintainability

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* improve fallback

Signed-off-by: pyranota <pyra@duck.com>

* remove comments from old migration

Signed-off-by: pyranota <pyra@duck.com>

* fix pull

Signed-off-by: pyranota <pyra@duck.com>

* fix once more

Signed-off-by: pyranota <pyra@duck.com>

* Update frontend/src/lib/components/ScriptBuilder.svelte

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* add sqlx cache

Signed-off-by: pyranota <pyra@duck.com>

* fix: frontend CI - fix debounce field names and remove leftover flow step debouncing

- Fix ScriptBuilder.svelte: change custom_debounce_key to debounce_key
- Add debounce_key and debounce_delay_s fields to NewScript schema in openapi.yaml
- Regenerate frontend types from OpenAPI spec
- Remove leftover flow step debouncing code from FlowModuleComponent.svelte
- Remove debounce fields from RawScript in openflow.openapi.yaml
- Remove unused Timer import from FlowModuleHeader.svelte

All frontend checks now passing (0 errors, 0 warnings)

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* fix ci

Signed-off-by: pyranota <pyra@duck.com>

* remove unused import

Signed-off-by: pyranota <pyra@duck.com>

* fix ci again

Signed-off-by: pyranota <pyra@duck.com>

* udpate ee repo ref

Signed-off-by: pyranota <pyra@duck.com>

* CI doesn't want to be fixed but I still try

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* ci...

Signed-off-by: pyranota <pyra@duck.com>

* Update ee-repo-ref.txt

* safer migration

Signed-off-by: pyranota <pyra@duck.com>

* reduce noise in logs

Signed-off-by: pyranota <pyra@duck.com>

* fix cli for scripts

Signed-off-by: pyranota <pyra@duck.com>

* nit

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Pyra <pyranota@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2025-10-23 20:18:08 +00:00
Pyra
caa21fcb33 fix: debounce_key automatic deletion (#6885) 
* fix: debounce_key automatic deletion

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* better solution

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2025-10-21 18:31:05 +00:00
Ruben Fiszel
f581ce6248 fix: monitor less frequent cleanup process periodicity 2025-10-17 18:27:11 +00:00
Pyra
defb6c9694 feat: dependency job debouncing (#6769) 
* v0

Signed-off-by: pyranota <pyra@duck.com>

* optimize relocks

* make it work with relative relative imports

Signed-off-by: pyranota <pyra@duck.com>

* use fallback

Signed-off-by: pyranota <pyra@duck.com>

* remove dbg and todos

Signed-off-by: pyranota <pyra@duck.com>

* future proof a bit

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* more cleanup

Signed-off-by: pyranota <pyra@duck.com>

* remove final TODO

Signed-off-by: pyranota <pyra@duck.com>

* do not use bytemuck

Signed-off-by: pyranota <pyra@duck.com>

* optimize hashing

Signed-off-by: pyranota <pyra@duck.com>

* implementation 1

Signed-off-by: pyranota <pyra@duck.com>

* almost v0

Signed-off-by: pyranota <pyra@duck.com>

* v0

Signed-off-by: pyranota <pyra@duck.com>

* add comments and use fallback

Signed-off-by: pyranota <pyra@duck.com>

* call dissolve for apps

Signed-off-by: pyranota <pyra@duck.com>

* add comms

Signed-off-by: pyranota <pyra@duck.com>

* refactor v0 (partially tested + dirty)

Signed-off-by: pyranota <pyra@duck.com>

* finishing

Signed-off-by: pyranota <pyra@duck.com>

* remove TODO

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* silence unused argument

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* implement rebuild_map endpoint

Signed-off-by: pyranota <pyra@duck.com>

* update windmill api client

Signed-off-by: pyranota <pyra@duck.com>

* almost finish with tests

Signed-off-by: pyranota <pyra@duck.com>

* add proper testing

Signed-off-by: pyranota <pyra@duck.com>

* remove unused fixtures

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* partial cleanup

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-worker/src/scoped_dependency_map.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/scripts.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* evil doings

Signed-off-by: pyranota <pyra@duck.com>

* more cleanup

* Update SQLx metadata

* more cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fixing CI

Signed-off-by: pyranota <pyra@duck.com>

* remove python from default features

Signed-off-by: pyranota <pyra@duck.com>

* feat: dependency job debouncing

* checkpoint

Signed-off-by: pyranota <pyra@duck.com>

* more improvements

Signed-off-by: pyranota <pyra@duck.com>

* refactor: clean up dependency job debouncing implementation

- Add comprehensive comments explaining the debouncing mechanism
- Replace debug statements (dbg!) with proper tracing calls
- Extract helper functions to reduce code duplication:
  - extract_to_relock_from_args() for extracting nodes/components
  - accumulate_debounce_stale_data() for updating stale data
- Improve code readability and maintainability

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* Update backend/windmill-common/src/jobs.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/jobs.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-queue/src/jobs.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* better error handling for helper

Signed-off-by: pyranota <pyra@duck.com>

* update ee-ref

Signed-off-by: pyranota <pyra@duck.com>

* test: add race condition test for dependency job debouncing

- Implement test_2 for scripts to test the race condition edge case
- Add comprehensive documentation comments to all test functions
- Remove empty test_2 stubs for flows (not needed)
- Keep test_2 stub for apps with TODO comment

The race condition test simulates the scenario where a job is marked
as running but debounce_key hasn't been cleaned up yet, forcing the
system to create a new job while reusing the existing debounce_key.

This edge case can occur due to the lack of transactions in the pull
function for performance reasons (see jobs.rs:4415-4425).

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* test: implement race condition test for dependency job debouncing

- Add comprehensive test_2 for script module that tests the race condition edge case
- Remove empty test_2 stubs from flows and apps modules
- Fix unused variable warning in worker_lockfiles.rs
- Add detailed comments explaining the race condition scenario and test logic

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* implement fallback

Signed-off-by: pyranota <pyra@duck.com>

* make it mostly work

* all tests are almost working

Signed-off-by: pyranota <pyra@duck.com>

* add comments a bit

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-common/src/scripts.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/worker_lockfiles.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/worker_lockfiles.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* feat: improve debouncing documentation, tracing, and error handling

- Add comprehensive 3-phase debouncing documentation explaining push/accumulation/pull
- Enhance all tracing with structured logging (job_id, workspace_id, node_count, etc.)
- Add proper error handling with .map_err() and contextual messages
- Replace dbg!() with proper tracing::debug!()
- Replace todo!() with proper error handling
- Fix typos: 'and edge case' → 'an edge case', 'bc' → 'because'
- Fix debug variable name: 'debounce_job_id_0' → 'debounce_job_id_o'
- Add documentation for debounce cleanup and stale data retrieval
- Add trace-level logging for non-error paths to reduce noise

Co-authored-by: Pyra <pyranota@users.noreply.github.com>

* do some work for future improvements

Signed-off-by: pyranota <pyra@duck.com>

* fix tests

Signed-off-by: pyranota <pyra@duck.com>

* clippy

Signed-off-by: pyranota <pyra@duck.com>

* update sqlx

Signed-off-by: pyranota <pyra@duck.com>

* clippy

Signed-off-by: pyranota <pyra@duck.com>

* update ee ref

Signed-off-by: pyranota <pyra@duck.com>

* flag tests behind the feature, add timeout

Signed-off-by: pyranota <pyra@duck.com>

* fix timeout + cleanup

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* row lock debounce_key

Signed-off-by: pyranota <pyra@duck.com>

* addressing TODOs

Signed-off-by: pyranota <pyra@duck.com>

* fix test

Signed-off-by: pyranota <pyra@duck.com>

* ee ref

Signed-off-by: pyranota <pyra@duck.com>

* ee repo

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Pyra <92104930+pyranye@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Pyra <pyranota@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-10-16 16:49:35 +00:00
Ruben Fiszel
aeb6829011 use excluded where relevant 2025-10-16 09:19:44 +00:00
hugocasa
e1d2eb2870 feat: support for protected private hub (#6762) 
* feat: support for protected private hub

* nits

* nits
 2025-10-08 09:33:28 +00:00
Alexander Petric
ae8d37fc34 feat: add dynamic skip for schedules (#6739) 
* feat: add dynamic skip for schedules

* npm check

* accidental rename

* fixing tests

* Update SQLx metadata

* simplify api / queries

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2025-10-08 09:18:15 +00:00
hugocasa
4bbbeb956f feat: powershell private repo support (#6684) 
* feat: powershell private repo support

* nits
 2025-09-29 13:24:41 +00:00
Ruben Fiszel
d56dea4969 fix: show more autoscaling events (#6704) 
* all

* all
 2025-09-29 13:24:10 +00:00
hugocasa
8f4fef9804 feat(backend): job result stream optimization (#6673) 
* feat(backend): job result stream optimization

* get offset locally instead of from db

* fix: agent worker result stream

* update ref

* nit

* remove foreign key on job

* fix build
 2025-09-26 11:26:50 +00:00
Ruben Fiszel
30b4ef9044 nit compile 2025-09-22 16:22:03 +00:00
Ruben Fiszel
f6dd78cb11 fix: improve reliability of exits in case graceful handler didn't exit as expected 2025-09-22 10:27:57 +00:00
Ruben Fiszel
56cdb69e59 fix: add termination handler earlier in lifecycle 2025-09-20 00:22:15 +00:00
dieriba
e98bde6be6 feat: allow operator to use script/flow with dynselect input (#6616) 
* done

* update sqlx

* fix openapi spec
 2025-09-17 10:56:39 +00:00
centdix
20f48e6ded feat(ai agent): handle images in ai agent (#6572) 
* add in frontend

* draft openai handling

* upload to s3

* simpler output

* return s3 directly if any

* low quality

* implement for gemini

* handle imagen model

* handle image input

* cleaning

* remove base64 from output

* cleaning

* fix timeout

* handle openrouter

* remove log

* allow image input when creating image

* cleaning

* increase stack size

* inline everything

* revert stack size

* cleaning

* fix for openai

* better mime type

* add descriptions
 2025-09-15 07:51:09 +00:00
Ruben Fiszel
6b34123aee fix: add OTEL_ENVIRONMENT to force otel environment 2025-09-09 14:21:06 +00:00
claude[bot]
20f4086a77 feat: add worker_group_job_stats table for efficient job metrics aggregation (#6527) 
* feat: add worker_group_job_stats table for job metrics aggregation

- Add new table with hour timestamp, worker group, script lang, workspace_id, job count and total duration
- Workers accumulate stats in memory and update hourly via sum aggregation
- Monitor.rs cleans up rows older than 60 days periodically
- Stats are flushed on worker shutdown to prevent data loss

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* all

* all

* ee-repo-ref

* nits

* nits

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-09-08 17:27:58 +00:00
Ruben Fiszel
d3f6a0da7e test logs 2025-09-02 21:32:21 +00:00
Ruben Fiszel
593da3213f add variable caching api 2025-09-02 14:34:46 +00:00
dieriba
f3fd1e90b0 feat: refactor trigger crud (#6472) 
* base

* base crate

* websocket updated

* crud refactored

* fix and nits

* fix compiler warning, nits and update repo ref

* fix oss

* fix compilation

* update ref

* fix

* update feature

* listener base

* add listener

* refactor logic done and implemented for postgres

* fix capture

* websocket

* implem for all triggers

* update sqlx,repo ref and fix

* fix oss

* unify struct fix websocket

* nits and fix oss runtime axum error

* perf cache query

* update .sqlx

* update .sqlx

* fix

* fix unused

* fix

* fix

* nits and fix http handler update endpoint

* update .sqlx

* update repo ref

* nits

* fix

* update ref

* fix

* update .sqlx
 2025-09-02 11:28:06 +00:00
claude[bot]
db36a323a7 feat(perf): add 60-second local cache for variables and resources(#6511) 
* feat: add 60-second cache for variables and resources with PostgreSQL invalidation

- Add new var_resource_cache module with 60-second TTL
- Implement PostgreSQL NOTIFY/LISTEN for immediate cache invalidation
- Cache get_variable() for non-secret, non-encrypted reads
- Cache get_resource_value() for all read operations
- Add database triggers on variable/resource table changes
- Initialize cache system in main.rs after database connection
- Add Clone derive to ListableVariable for cache compatibility

Performance benefits:
- Avoids database queries on cache hits
- Immediate invalidation ensures data consistency
- Selective caching respects security constraints

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: move cache to API layer with allow_cache query param

- Move variable/resource cache from windmill-common to windmill-api
- Add allow_cache query parameter to variables and resources endpoints
- Follow raw script cache pattern with timestamp + value structure
- Create proper database migration for notification triggers
- Include encrypted values in cache when appropriate
- Only activate caching when explicitly requested

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* refactor: move cache to API layer with allow_cache query param

Move PostgreSQL LISTEN logic to main.rs following established pattern
- Remove custom listener initialization from cache module
- Add cache invalidation channels to centralized notification handler
- Simplified cache module to only handle cache operations
- Follow raw script cache pattern for notification handling

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
 2025-09-02 07:48:03 +00:00
Ruben Fiszel
5510b1dd95 typo env nit 2025-08-31 12:23:37 +00:00
claude[bot]
8f46066a64 feat(perf): add 300-second local cache for variable crypt retrieval (#6483) 
* feat: add 60-second cache for workspace key retrieval

This implements a cache with 60-second staleness for the get_workspace_key
function to reduce database queries for workspace encryption keys. The cache
follows the same pattern as the existing CUSTOM_ENVS_CACHE but with a
shorter expiration time.

Requested by @rubenfiszel

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* Extend cache staleness from 60 to 300 seconds

* feat: add cache invalidation notifications for workspace keys

Add PostgreSQL LISTEN/NOTIFY mechanism to invalidate workspace key cache
across all servers and workers when workspace keys change.

- Add database migration with trigger function for workspace_key changes
- Add notification handler in main.rs to remove from WORKSPACE_KEY_CACHE
- Follow same pattern as existing workspace environment cache invalidation
- Ensures distributed cache consistency for workspace encryption keys

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* finish

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
 2025-08-31 12:19:59 +00:00
hugocasa
2ce11cea3b prevent past due team plans to overuse (#6487) 
* prevent past due team plans to overuse

* nit

* update ee ref
 2025-08-28 16:56:57 +00:00
Ruben Fiszel
4ad0d255f3 feat: autovacuum or high intensity tables 2025-08-27 17:29:06 +00:00
Ruben Fiszel
99666426ec nits 2025-08-23 07:54:59 +00:00
claude[bot]
49ed757424 feat: add prometheus metric queue_running_count (#6413) 
* feat: add prometheus metric queue_running_count

Adds a new Prometheus metric queue_running_count that tracks the number
of currently running jobs per tag, similar to the existing queue_count
metric but filtered for running=true instead of running=false.

Changes:
- Added get_queue_running_counts() function in windmill-common/src/queue.rs
- Added QUEUE_RUNNING_COUNT Prometheus metric in monitor.rs
- Added /workers/queue_running_counts API endpoint
- Updated OpenAPI specification
- Added SQLx query cache entry

Requested by @rubenfiszel

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* Remove database insertion for queue_running_count metrics

Keep Prometheus metrics and API endpoint functionality while removing
the database INSERT statements as requested.

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* sqlx

* improve logic

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-08-19 22:12:36 +00:00
pyranota
11dd4118ce feat: add ruby support (#5939) 
* implement single line pin

* make panic-safe

* use pin even if multiple modules imported withing single statement

* add repins and make imports respect pins

* keep all pins

* Allow multiple pins

* add comments + handle stuff more safely

* fix fully qualified imports

* remove ignore

* sort nested

* apply unique to output requirements list

* fix typo

* remove mut

* update sqlx

* sort imports

* sort imports

* fix formatter and format

* feat: add ruby support

* fix

* update sqlx

* add ruby to flake.nix

Signed-off-by: pyranota <pyra@duck.com>

* upgrade par_install_language_dependencies helper

rename fields + allow passing custom payload

Signed-off-by: pyranota <pyra@duck.com>

* change migrations

Signed-off-by: pyranota <pyra@duck.com>

* make full pipeline work!

Signed-off-by: pyranota <pyra@duck.com>

* do a bit fixing

* stage

Signed-off-by: pyranota <pyra@duck.com>

* add ruby in dockerfiles

Signed-off-by: pyranota <pyra@duck.com>

* implement auto-require + 'windmill/inline'

Signed-off-by: pyranota <pyra@duck.com>

* create 'windmill/inline' only if does not exist

Signed-off-by: pyranota <pyra@duck.com>

* add nsjail

Signed-off-by: pyranota <pyra@duck.com>

* update global settings

Signed-off-by: pyranota <pyra@duck.com>

* add ruby icon

* generate lockfile on deploy (but broken)

Signed-off-by: pyranota <pyra@duck.com>

* fix(frontend): display deployed script lockfile and other info

Signed-off-by: pyranota <pyra@duck.com>

* feat: add mini-wmill client

Methods:
- get_resource
- get_variable

Additionally add shortcuts on frontend

Signed-off-by: pyranota <pyra@duck.com>

* add nit warning

Signed-off-by: pyranota <pyra@duck.com>

* support private repos credentials

Signed-off-by: pyranota <pyra@duck.com>

* fix compilation error

Signed-off-by: pyranota <pyra@duck.com>

* found rustc bug + refactor universal installer

Signed-off-by: pyranota <pyra@duck.com>

* more refactor + fixes

Signed-off-by: pyranota <pyra@duck.com>

* bug fixing + polishing

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* change 'windmill/inline'

Signed-off-by: pyranota <pyra@duck.com>

* delegate env_clear to universal installer

Signed-off-by: pyranota <pyra@duck.com>

* update init script

Signed-off-by: pyranota <pyra@duck.com>

* add cli support

Signed-off-by: pyranota <pyra@duck.com>

* unhardcode frontend

Signed-off-by: pyranota <pyra@duck.com>

* fix cli

Signed-off-by: pyranota <pyra@duck.com>

* fix init script

Signed-off-by: pyranota <pyra@duck.com>

* make it work on ms windows

Signed-off-by: pyranota <pyra@duck.com>

* do not clean env in the end

Signed-off-by: pyranota <pyra@duck.com>

* fix windows (again)

Signed-off-by: pyranota <pyra@duck.com>

* include PROXY_ENVS (untested)

Signed-off-by: pyranota <pyra@duck.com>

* feat: self-signed certs + proxy support (tested)

Signed-off-by: pyranota <pyra@duck.com>

* flake: pin ruby to 3.4

* merge follow-ups

Signed-off-by: pyranota <pyra@duck.com>

* remove obsolete libc crate

Signed-off-by: pyranota <pyra@duck.com>

* cleanup parser

Signed-off-by: pyranota <pyra@duck.com>

* fix java ci

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fix nits

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-worker/src/universal_pkg_installer.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/ruby_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* add nsjail for lock stage

Signed-off-by: pyranota <pyra@duck.com>

* fix merge

Signed-off-by: pyranota <pyra@duck.com>

* Update common.rs

* regen windmill-api-client

Signed-off-by: pyranota <pyra@duck.com>

* fix tests

Signed-off-by: pyranota <pyra@duck.com>

* fix tests again

Signed-off-by: pyranota <pyra@duck.com>

* reference published ruby parser

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
 2025-08-15 13:03:32 +00:00
Ruben Fiszel
65bcc00cd9 fix: add timeouts to more queries to prevent some rare deadlocks scnarios 2025-08-14 17:31:38 +00:00
Ruben Fiszel
147e6975c4 feat: add configurable stale jobs detection and cancellation 2025-08-08 17:32:11 +00:00
dieriba
414f099188 feat: add instance-wide workspace prefix option for custom app (#6180) 2025-08-07 17:55:26 +00:00
claude[bot]
161dbbc6d6 feat: add native result streaming (#6242) 
* feat: add stream output feature to SSE job updates

Adds stream_output field to JobUpdate struct that extracts log lines
starting with '[wm_stream]:' from job logs. Regular logs now exclude
stream lines, which are captured separately for specialized handling.

- Added stream_output: Option<String> field to JobUpdate struct
- Created extract_stream_output_from_logs() function to filter stream lines
- Modified get_job_update_data() to use stream extraction logic
- SSE clients now receive both new_logs and stream_output in job updates

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* feat: rename stream_output to stream and handle newlines as \n

- Renamed `stream_output` field to `stream` in JobUpdate struct
- Updated extract_stream_output_from_logs to extract_stream_from_logs
- Changed stream output to join with literal \n instead of actual newlines
- Stream lines are properly excluded from regular new_logs

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* decision tree nits

* push ee ref

* push ee ref

* fix: fix id renaming in apps

* remove duplicate caching (#6285)

* feat: migrate audit log ids to bigints (blocking migration for EE)

* fix(mcp): add proper check for mcp routes (#6282)

* add proper check for mcp routes

* cleaner

* apply to flow

* fix add checks scopes

---------

Co-authored-by: dieriba <dieriba.pro@gmail.com>

* chore(main): release 1.514.0 (#6283)

* chore(main): release 1.514.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* fix: pin tokio to 1.46.1 and aws-sdks-ts

* pin rustls to 0.23.29 + pin aws-sdk

* chore(main): release 1.514.1 (#6288)

* chore(main): release 1.514.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* fix: improve docker logs collection in docker mode

* support $res: string in form inputs of arrays

* fix import nit

* fix: fix DynSelect

* nits

* fix: resource-type-ts-parser (#6289)

* fix: resource types as arg in typescript handle imported defined types

* Update nix flake (#6291)

* merge

* Small UI fixes (#6294)

* fix step history not refreshing with staticInputs

* fix array of obj not showing up in json editor in test this step

* datatable scales correctly in DisplayResult and scrolling is much more usable

* avoid next button disapearing and changing layout / hurting ux

* nits

* fix bug when renaming module A to B then module C to A, C takes the schema of A

* fix bug with comments in sql repl

* fix aggrid theme randomly not loading

* bindable script

* better delete button in db manager

* property select doesnt exist

* fix all warnings

* delete $flowStateStore[id] on delete

* feat(cli): generate cursor rules on init (#6270)

* create cursor rules on init

* change gen

* add missing resource-type command

* add resource type command in guidance

* add schema option

* revert

* nit

* nit

* add flow guidance

* nit

* chore(main): release 1.515.0 (#6292)

* chore(main): release 1.515.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* fix: improved logs for script

* nits logs

* chore(main): release 1.515.1 (#6295)

* chore(main): release 1.515.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* merge

* even more indexer tracings

* add more tracing logs

* feat: prevent too large results (>500Mb) from OOMing database

* nit naming

* feat: add CA certificate update at startup via environment variable (#6280)

* feat: add CA certificate update at startup via environment variable

Add support for running 'update-ca-certificates' at binary startup
when RUN_UPDATE_CA_CERTIFICATE_AT_START environment variable is set to "true".

- Check for RUN_UPDATE_CA_CERTIFICATE_AT_START env var on startup
- Execute update-ca-certificates command if env var is set to "true"
- Log success/failure appropriately with tracing
- Continue startup even if CA certificate update fails
- Non-blocking implementation with proper error handling

Fixes #6279

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* refactor: extract CA certificate update logic into separate function

Extract the CA certificate update logic from windmill_main() into a
dedicated update_ca_certificates_if_requested() function for better
code organization and maintainability.

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* improvements

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com>
Co-authored-by: Alexander Petric <alex@windmill.dev>

* fix: indexer collection of job logs before indexing (#6300)

* Add flume as dependecy for indexer

* Update ee-repo-ref

* Remove flags from cargo.toml

* Update ee-repo-ref

* Update ee-repo-ref

* fix rust sdk build error (#6305)

Signed-off-by: pyranota <pyra@duck.com>

* fix broken audit logs filter (#6304)

* rename to from to

* goto fix

* default to false if field not present operator settings (#6301)

* git sync UI improvements (#6303)

* ui improvements round 1

* modal cleanup

* init

* UI refactor

* UI cleanup + refactor

* legacy cleanup

* success model -> github actions, non-ee warnings

* sqlx

* npm check

* ee warning everywhere

* last comments

* formatting

* no hardcoded theme

* claude review improvemenets

* fix: no process relative imports for scripts with codebase

* fix: sqs oidc authentication disconnect #6307

* handle metadata for new scripts happen after commit

* handle_deployment_metadata in a task

* nits

* chore: add windmill-utils-internal package (#6299)

* add utils package

* naming

* cleaning

* add docs

* remove log

* use autogenerated types

* remove old

* fix

* cleaning

* add docs

* chore(main): release 1.516.0 (#6298)

* chore(main): release 1.516.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* merge

* indexer improvements

* upgrade tantivy to 0.24.2

* use tantivy fork

* nit warnings

* fix oss build

* improve indexer

* chore: use windmill-utils-internal for cli (#6297)

* add utils package

* naming

* cleaning

* simplify assignPath

* rename old files

* same for locks

* create on confirm

* default true

* use replaceinlinescripts from utils

* use extractscriptfromflows

* make it compile

* cleaning

* use argsigtojson

* fix

* fix missing await

* cleaner

* cleaning

* cleaning

* use in frontend

* add docs

* testing

* remove log

* use autogenerated types

* remove old

* fix

* cleaning

* adapt usage

* draft

* better build script

* fix build

* revert to default creation

* add docs

* remove and rename

* make everything work

* add await

* only if not installed

* add vs code setting

* add to publish action

* fix bc

* safer use of sep

* fix

* do not rename on push

* no publish on release

* use published package on frontend

* nit

* Add dependencies to run sqlx prepare to nix flake (#6309)

* feat(cli): wmill-lock.yaml v2 for easier git merge diffs

* merge

* merge

* all

* all

* rm warnings

* fix styling on aichatinput (#6312)

* fix: use with_capacity back presusre for tantivy directory multipart writes (#6313)

* use with capacity for tantivy directory multi part uploads

* Update ee repo ref

* Update ee-repo-ref

* Update ee-repo-ref

* chore(main): release 1.517.0 (#6310)

* chore(main): release 1.517.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* fix typo on cli build (#6314)

* cleanup

* feat(utils): add flow.yaml validation function (#6316)

* add validateflow function

* cleaner code

* preprocess json

* cleaning

* create specific package

* cleaning

* add tests

* fix: cleanup concurrency_counter automatically + remove orphans keys automatically

* fix: add disabled support to resource picker in schema forms

* fix: add wm_labels to tracing spans

* all

* merge

* all

* fix: delete empty git connection (#6318)

* fix checks

* bun handling

* all

* all?

* all

* all

* update

* all

* update

* check

* fix history

* all

* all

* all

* Remove leftover debug tracing statements

- Remove commented debug trace in jobs.rs for stream output
- Remove commented debug trace in result_stream.rs for stream processing

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* fix test

* all

* handle iter

* fix

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
Co-authored-by: dieriba <dieriba.pro@gmail.com>
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
Co-authored-by: wendrul <53628737+wendrul@users.noreply.github.com>
Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com>
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com>
Co-authored-by: Alexander Petric <alex@windmill.dev>
Co-authored-by: pyranota <92104930+pyranota@users.noreply.github.com>
 2025-08-06 22:40:42 +00:00
Ruben Fiszel
87dd52296f fix: cleanup concurrency_counter automatically + remove orphans keys automatically 2025-08-01 11:26:26 +00:00
claude[bot]
a460e131c7 feat: add CA certificate update at startup via environment variable (#6280) 
* feat: add CA certificate update at startup via environment variable

Add support for running 'update-ca-certificates' at binary startup
when RUN_UPDATE_CA_CERTIFICATE_AT_START environment variable is set to "true".

- Check for RUN_UPDATE_CA_CERTIFICATE_AT_START env var on startup
- Execute update-ca-certificates command if env var is set to "true"
- Log success/failure appropriately with tracing
- Continue startup even if CA certificate update fails
- Non-blocking implementation with proper error handling

Fixes #6279

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* refactor: extract CA certificate update logic into separate function

Extract the CA certificate update logic from windmill_main() into a
dedicated update_ca_certificates_if_requested() function for better
code organization and maintainability.

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* improvements

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com>
Co-authored-by: Alexander Petric <alex@windmill.dev>
 2025-07-30 14:39:26 +00:00
dieriba
c4178c05be feat: periodic worker group script (#6159) 
* feaat periodic script

* feat periodic script

* fix unused

* check minimun time

* fix unused import

* update repo ref

* update ref

* nits and update repo ref

* add missing checks

* fix

* improve UX

* improve UX

* improve UX

* fix

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-07-21 22:57:56 +00:00
dieriba
dd1f520d83 fix: improve ssh agent worker naming (#6211) 
* remove ssh suffix

* use hostname tags

* feat

* feat

* update sqlx

* use hostname

* use worker prefix

* nits
 2025-07-21 13:43:25 +00:00
Ruben Fiszel
dec72e201b feat: use sse for flow status updates 2025-07-19 14:42:26 +00:00
Ruben Fiszel
ce442a7493 nit 2025-07-13 23:02:42 +00:00
Ruben Fiszel
b4a8d7212b nit unused import 2025-07-13 22:25:05 +00:00
Ruben Fiszel
372d56ae40 apply iptables for cloud hosted only 2025-07-13 20:08:09 +00:00
wendrul
835645643e feat: Better tracing for audit logs, including a graph to visualize them (#6078) 
* Migrate audit log page to svelte 5

* Add email and span cols to audit table

* Add token_prefixs to audit logs (into AuditAuthorable trait)

* Add audit logs graph (wip)

* Add audit span on push and jwt

* Unify same job audit into the same audit span

* Improve the graph visually

* Fix typo

* functioning graph with svelte issue

* Fix leak

* feat: migrate AuditLogsTable from DataTable to VirtualList for performance

- Replace DataTable component with VirtualList for handling thousands of rows
- Migrate to Svelte 5 runes ($props, $bindable, $derived, $state)
- Implement flattenLogs() for virtual scrolling with grouped date headers
- Add sticky indices and dynamic height calculation
- Update parent component to use callback prop pattern instead of events
- Preserve all existing functionality: filtering, selection, pagination
- Follows RunsTable.svelte implementation pattern

Resolves performance issues when displaying large audit log datasets.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* Fix remaining virtual list issues

* WIP graph

* Fix chart styling

* Fix npm check

* Fix missing audit_span arguments

* Update sqlx

* use varchar 255 for email as in other tables

* Remove syntax inconsistency

* Match struct with ee crate

* Update ee-repo-ref.txt

* Update worker_flow.rs

* Remove redefinition of trait to prevent shadowing

* Re add trait on oss but only when no `private` flag

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
 2025-07-03 21:02:45 +00:00
Ruben Fiszel
86eb9074cc feat: blacklist remote agent worker token (#5985) 2025-06-20 12:07:33 +02:00
Ruben Fiszel
126b610561 introduce QUIET_LOGS 2025-06-13 11:25:27 +02:00
pyranota
22b2f4988d fix: never consider minor version for global site packages (#5893) 2025-06-07 03:35:28 +02:00
Alexander Petric
6d3a2d279a invalidate auth cache when logging out (#5869) 
* invalidate auth cache when logging out

* use proper sqlx migrate add

* formatting

* pg notify on token, not email
 2025-06-05 09:30:23 +02:00
dieriba
f2526571a3 feat: ssh repl like direct to workers hosts machine (#5809) 
* feat(backend): add repl for worker

* feat(frontend): add repl to interact with worker and missing packages

* nits

* Update backend/windmill-worker/src/result_processor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/result_processor.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/WorkerRepl.svelte

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* fix typo

* fix cd

* nits

* refactoring

* feat: handle other agent worker to also access repl feature, fix bugs

* update repo ref

* nits and match function args

* nits

* typo

* remove struct AgentWorkerData

* update repo ref

* nits

* impl new for authed client and revert to async

* update ref

* updage ee repo ref

* fix missing method/imports small bugs

* update ee repo ref

* update .sqlx

* test

* clean wait for interactive_shell future

* free call stack

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
 2025-06-03 18:41:14 +02:00
Diego Imbert
0e316239dd EE Refactor (#5844) 
* app compiles with every ee substituted

* Replace all oss files content

* Revert "Replace all oss files content"

This reverts commit ea4017d59f.

* delete all ee

* hide all _ee files under private flag

* hide every oss stuff when private flag set

* pub use *

* gitignore and substitute script

* pub mod for ee needed for ee repo

* small mistakes

* remove oidc_oss impl

* ee ref (temp)

* ee ref

* fix --all-features selecting private in OSS CI

* ee repo ref

* allow unused
 2025-06-02 22:12:33 +02:00
Ruben Fiszel
5b96bccedd feat: add aws oidc support for instance s3 storage (#5810) 
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
 2025-05-25 14:03:38 +02:00
HugoCasa
30edcdfe0e fix: workspace preprocessor improvements (#5784) 
* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix: workspace preprocessor fixes

* tmp ee ref

* fix build

* update ee ref

* fix: hub script preprocessor handling

* fix build

* good ref

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
 2025-05-21 17:49:21 +02:00
Ruben Fiszel
7a43893616 nit 2025-05-20 13:23:02 +02:00