Commit Graph

1640 Commits

Author SHA1 Message Date
Ruben Fiszel
81ff0dcd8c fix: retry python relative imports on errno 104 2025-09-20 13:41:15 +00:00
Ruben Fiszel
1afb7a2156 improve job cancelling with new flow jobs locks 2025-09-20 11:00:45 +00:00
Ruben Fiszel
9a21107848 improve job cancelling with new flow jobs locks 2025-09-20 10:17:10 +00:00
Ruben Fiszel
88b4bc2c9b remove tracing err 2025-09-20 08:11:19 +00:00
Ruben Fiszel
9d8641cdd0 fix flow jobs 2025-09-20 08:07:41 +00:00
Ruben Fiszel
fb93024d3e vacuum skip_locked 2025-09-20 00:42:43 +00:00
Ruben Fiszel
56cdb69e59 fix: add termination handler earlier in lifecycle 2025-09-20 00:22:15 +00:00
hugocasa
d365014882 build on macos with deno_core (#6642) 2025-09-19 17:33:05 +00:00
Ruben Fiszel
f71f9b0894 feat: load for loop jobs timeline directly from for loop flow status (#6646) 2025-09-19 17:32:09 +00:00
hugocasa
993baf46bd feat(backend): flow streaming (#6520)
* feat(backend): flow streaming

* all streaming languages + sync api

* sqlx

* fix build

* UI and nits

* nit

* feat: stream last flow step

* sqlx

* nit

* use get for stream endpoints + add snippet in UI

* refactor

* nits

* Update backend/windmill-worker/src/common.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nits

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-09-18 15:49:35 +00:00
centdix
709a937ac2 fix(aiagent): fix endpoint for azure (#6633)
* fix ai agent for azure

* cleaning

* add parentheses
2025-09-18 13:22:24 +00:00
centdix
3199f9fffd feat(ai agent): allow multiple images input for ai agent + code cleaning (#6591)
* add in frontend

* draft openai handling

* upload to s3

* simpler output

* return s3 directly if any

* low quality

* implement for gemini

* handle imagen model

* handle image input

* cleaning

* remove base64 from output

* cleaning

* fix timeout

* handle openrouter

* remove log

* allow image input when creating image

* cleaning

* increase stack size

* inline everything

* revert stack size

* refactor: move AI executor types to separate module

- Created ai module structure with types.rs
- Moved all type definitions from ai_executor.rs to ai/types.rs
- No functional changes, just code organization

* refactor: add QueryBuilder trait and provider detection utilities

- Created QueryBuilder trait for abstracting provider-specific logic
- Added helper functions for provider detection (is_anthropic_provider)
- Implemented placeholder QueryBuilder for all providers
- Updated OpenAIRequest to use slices instead of Vec references
- All providers now have QueryBuilder implementations (using default for now)

* feat: implement OpenAI query builder with image support foundation

- Created proper OpenAI query builder implementation
- Added image_handler module for S3 upload/download utilities
- Separated text and image request building logic
- Added prepare_messages_for_api to handle S3Object conversion
- Foundation laid for supporting tools with image output

* refactor(ai): complete AI executor refactoring with query builder pattern

- Created modular structure under ai/ module
- Moved all types to ai/types.rs
- Created QueryBuilder trait for provider abstraction
- Implemented OpenAI query builder with image+tools support
- Added unified agent runner supporting both text and image outputs with tools
- Refactored run_agent to delegate to new unified implementation
- Added image handler utilities for S3 operations
- Improved code organization and maintainability

* cleaning

* feat(ai): implement remaining provider query builders

- Added Anthropic query builder with proper message conversion
- Added Google AI query builder with Gemini API support
- Added OpenRouter query builder delegating to OpenAI for compatibility
- Added missing Anthropic and Gemini types to types.rs
- Fixed type references and compilation errors
- All providers now support the unified query builder interface

* fixes

* fixes

* mime type + cleaning

* image to images

* handle mutlitple images

* fix

* remove agent_runner file

* clean query builder logic

* cleaning

* cleaning

* hide structured_output based on output type

* fix

* user images and not nested

* better descriptions
2025-09-17 16:55:54 +00:00
wendrul
9325f5636c fix: Don't reencrypt secrets on workspace forking (#6622)
* Do not reencrypt variable when performing a workspace fork

* Update SQLx metadata

* Update SQLx metadata

* Remove unused

---------

Co-authored-by: GitHub Action <action@github.com>
2025-09-17 16:47:27 +00:00
hugocasa
c24c629317 fix: run preprocessor even if empty flow (#6609) 2025-09-15 12:13:43 +00:00
centdix
20f48e6ded feat(ai agent): handle images in ai agent (#6572)
* add in frontend

* draft openai handling

* upload to s3

* simpler output

* return s3 directly if any

* low quality

* implement for gemini

* handle imagen model

* handle image input

* cleaning

* remove base64 from output

* cleaning

* fix timeout

* handle openrouter

* remove log

* allow image input when creating image

* cleaning

* increase stack size

* inline everything

* revert stack size

* cleaning

* fix for openai

* better mime type

* add descriptions
2025-09-15 07:51:09 +00:00
Ruben Fiszel
aa68ac8336 more tracing on archiving scripts 2025-09-11 17:34:54 +00:00
pyranota
c4ccc4ba14 fix(apps): fix relative imports cache invalidation (#6564)
* v0

Signed-off-by: pyranota <pyra@duck.com>

* optimize relocks

* make it work with relative relative imports

Signed-off-by: pyranota <pyra@duck.com>

* use fallback

Signed-off-by: pyranota <pyra@duck.com>

* remove dbg and todos

Signed-off-by: pyranota <pyra@duck.com>

* future proof a bit

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* more cleanup

Signed-off-by: pyranota <pyra@duck.com>

* remove final TODO

Signed-off-by: pyranota <pyra@duck.com>

* do not use bytemuck

Signed-off-by: pyranota <pyra@duck.com>

* optimize hashing

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-09-11 10:46:01 +00:00
claude[bot]
fcd58191d4 fix: force stdin to Stdio::null for all user code execution (#6575)
Set stdin to Stdio::null for all Commands that execute user code across all supported languages to prevent unwanted input consumption. This affects Python, Deno, Bash, PowerShell, Go, Rust, PHP, Ruby, Java, C#, Ansible, Nu, and Bun executors.

The dedicated worker handler was intentionally left unchanged as it requires stdin for inter-process communication.

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2025-09-10 16:14:25 +00:00
Diego Imbert
304f953710 Separate duckdb crate to fix c++ build/link issues (#6551)
* call ffi

* remove duckdb dep

* rename windmill_duckdb_ffi_internal

* static lib

* ci

* back to dylib, bug isn't fixed in static

* feature flag and copy dynamic lib

* fix dynlib in docker

* load libwindmill_duckdb_ffi_internal at runtime on usage

* lazy static deadlocks

* Cache dynamic library handles

* update auto s3 path insert from editor bar

* Fix duckdb S3 freezing worker because of blocking task in tokio async context

* build dll windows GH workflow

* try fix windows build

* revert build.rs

* nit fixes CI

* Dockerfile update (not tested yet

* build dev sh for duckdb lib

* mistake

* attach windmill_duckdb_ffi_internal.so artefact

* rhel9

* docker fixes

* fix dockerfile

* better err msg

* forgot lib prefix .so

* add column_order

* fix column_order
2025-09-10 14:43:46 +00:00
Ruben Fiszel
cf70265ee9 rust tests cleanup (#6573)
* test improvements

* improve tests

* improve tests
2025-09-10 14:43:10 +00:00
hugocasa
36bbde6239 feat: email triggers (#6548)
* feat: email triggers

* Change down migration to drop email_trigger table

* email triggers UI

* bug fix

* Apply suggestion from @ellipsis-dev[bot]

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* cli and git sync

* Revert "cli and git sync"

This reverts commit 220fd50d13.

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-09-09 21:04:23 +00:00
Ruben Fiszel
4f9e5badf9 fix: add request duration to relative imports loading in python 2025-09-09 09:09:33 +00:00
claude[bot]
20f4086a77 feat: add worker_group_job_stats table for efficient job metrics aggregation (#6527)
* feat: add worker_group_job_stats table for job metrics aggregation

- Add new table with hour timestamp, worker group, script lang, workspace_id, job count and total duration
- Workers accumulate stats in memory and update hourly via sum aggregation
- Monitor.rs cleans up rows older than 60 days periodically
- Stats are flushed on worker shutdown to prevent data loss

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* all

* all

* ee-repo-ref

* nits

* nits

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-09-08 17:27:58 +00:00
dieriba
c210a404e0 feat: timeout as expression (#6509)
* done

* add ctx and flow input

* backward compatible

* fix typo

* fix
2025-09-08 15:42:21 +00:00
Ruben Fiszel
83fff3a590 fix: fix db pool corruption by nativets logs 2025-09-06 14:15:18 +00:00
pyranota
f50ab6387a fix(flows): fix relative imports cache invalidation [ext] (#6546)
* fix: follow up on prev pr

Signed-off-by: pyranota <pyra@duck.com>

* Update worker_lockfiles.rs

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-09-05 17:07:13 +00:00
pyranota
d0982432e2 fix(flows): fix relative imports cache invalidation (#6519)
* fix(flows): fix relative imports cache invalidation

Signed-off-by: pyranota <pyra@duck.com>

* update flow_versions in the end

Signed-off-by: pyranota <pyra@duck.com>

* comms + fallback

Signed-off-by: pyranota <pyra@duck.com>

* add warn if fallback is used

Signed-off-by: pyranota <pyra@duck.com>

* include referenced scripts' hashes in flow_node id

Signed-off-by: pyranota <pyra@duck.com>

* improve fallback + TODOs

Signed-off-by: pyranota <pyra@duck.com>

* perf: optimize trigger

Signed-off-by: pyranota <pyra@duck.com>

* reduce TODOs

Signed-off-by: pyranota <pyra@duck.com>

* remove more TODOs

Signed-off-by: pyranota <pyra@duck.com>

* test thing

Signed-off-by: pyranota <pyra@duck.com>

* remove TODO

Signed-off-by: pyranota <pyra@duck.com>

* drop old flow trigger

Signed-off-by: pyranota <pyra@duck.com>

* add comments to migration

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-09-05 16:06:51 +00:00
centdix
832d120e6c chore: better error message for ai agent + add back .ts in utils-internal (#6544)
* better error message

* add back ext after publishing

* add back ext
2025-09-05 14:52:41 +00:00
Ruben Fiszel
1611404e95 fix flow push schedule on flow update (#6538)
* fix flow schedule

* fix flow schedule
2025-09-05 10:09:17 +00:00
centdix
c0dbbe942c feat(flows): allow all providers for ai agent steps (#6529)
* add ui for structured output

* implement backend logic for openai models

* simulate having non required props

* cleaning

* implement logic for anthropic

* cleaning

* cleaning

* cleaning

* avoid name clash

* return object

* focus new field when adding

* fix condition

* small opti

* use box raw value

* avoid unnecessary parsing + return error if parsing fails

* update ui to show all providers

* cleaning

* adapt backend

* fix

* styling fix

* fix for anthropic with openrouter

* nit styling

* avoid race condition

* fix merge

* fix

* fix import

* make one component for whole provider selection

* fix resource logic

* rename

* clean

* remove console log

* fix bad copy
2025-09-05 08:11:23 +00:00
dieriba
f3f330dd2a feat: windmill dyn multiselect (#6488)
* done

* fix

* better

* typo

* use old key

* chore: publish pkgs and update deps

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-09-05 07:10:01 +00:00
hugocasa
b69478cf2b feat: better early stop in flows (#6534)
* feat: better early stop

* style

* nits

* nits

* fix test
2025-09-05 06:25:53 +00:00
Ruben Fiszel
0647202a4f fix: check perms more tightly for running jobs (#6532)
* all

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* all

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* nits
2025-09-05 06:07:45 +00:00
dieriba
f10cac1c4b feat: conditional retry workflow (#6461)
* feat retry

* fix

* fix

* fix and nits

* fix query

* fix test

* nits and perf

* update sqlx

* once

* remove

* update sqlx
2025-09-03 20:24:38 +00:00
centdix
bf7ba698a1 feat(flow): add structured output option for ai agent step (#6515)
* add ui for structured output

* implement backend logic for openai models

* simulate having non required props

* cleaning

* implement logic for anthropic

* cleaning

* cleaning

* cleaning

* avoid name clash

* return object

* focus new field when adding

* fix condition

* small opti

* use box raw value

* avoid unnecessary parsing + return error if parsing fails
2025-09-03 16:43:53 +00:00
Diego Imbert
420be5a1f8 feat: use S3 Proxy for duckdb instead of direct connection (#6505)
* s3 proxy works with get (no auth yet)

* nit

* support s3:// syntax

* Support s3:// syntax and fix vite api proxy normalizing double slashes in URI

* s3 checks authed

* nit

* PUT works

* delete file works

* Derive the JWT signature from the backend

* Authorize s3 correctly (JWT signature is never sent in cleartext)

* convert object store error to wmill error for correct status code

* stash

* fix

* POST first request proxy works

* s3 put for duckdb

* factor out direct proxy code

* Fix Issue with backend proxy and wrong signature due to Host header mismatch

* Add _default_ syntax to solve URI normalization issues with signing

* restricted to user paths toggle

* user path restriction works !

* change restriction to allow

* fix

* factor out code

* better permissions UX in object storage settings

* Revert to restrict_to_user_paths

* check permissions in old s3 api

* DuckDB now uses S3 Proxy and no longer needs LFS query

* implement todo

* fix hardcoded w_id

* s3 proxy size limit

* s3_proxy is ee

* nit

* add Google Cloud Storage as option to secondary storage

* GCS secret in duckdb

* fix toolchain compile

* Remove user permissions for v0

* fix ci 2

* fix CI OSS

* fix missing feature flag

* fix unused warning

* integration test fails bc rustc 1.85.0

* ee ref

* fix ci ...

* update ee ref
2025-09-03 15:03:44 +00:00
claude[bot]
dea7c6da7e feat: add S3 streaming info to 10k rows SQL error message (#6528)
Extended the error message when SQL queries return more than 10k rows
to inform users about S3 streaming capability with a link to docs

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2025-09-03 13:18:23 +00:00
dieriba
f3fd1e90b0 feat: refactor trigger crud (#6472)
* base

* base crate

* websocket updated

* crud refactored

* fix and nits

* fix compiler warning, nits and update repo ref

* fix oss

* fix compilation

* update ref

* fix

* update feature

* listener base

* add listener

* refactor logic done and implemented for postgres

* fix capture

* websocket

* implem for all triggers

* update sqlx,repo ref and fix

* fix oss

* unify struct fix websocket

* nits and fix oss runtime axum error

* perf cache query

* update .sqlx

* update .sqlx

* fix

* fix unused

* fix

* fix

* nits and fix http handler update endpoint

* update .sqlx

* update repo ref

* nits

* fix

* update ref

* fix

* update .sqlx
2025-09-02 11:28:06 +00:00
hugocasa
fbd942f179 fix(backend): retrieve root_job for MiniPulledJob + fix root job for flow jobs (#6490)
* feat(backend): retrieve root_job for MiniPulledJob + fix root job for flow jobs

* nit

* nits

* set root_job only if additional information meaning if not equal to innermost or parent

* nit

* feat: add root job env var

* nits

* fix build

* nit

* fix build

* nit

* sqlx
2025-09-02 09:44:09 +00:00
Ruben Fiszel
16912b484d feat: cache relative imports (#6504)
* all

* all

* update
2025-09-01 14:55:08 +00:00
Ruben Fiszel
641d5651c5 fix: fix error handling of pre-processor steps 2025-08-29 09:38:25 +00:00
hugocasa
2ce11cea3b prevent past due team plans to overuse (#6487)
* prevent past due team plans to overuse

* nit

* update ee ref
2025-08-28 16:56:57 +00:00
pyranota
006f32602c fix: fix relative imports cache invalidation (#6468)
* fix: new script on deps job for secondary scripts

Signed-off-by: pyranota <pyra@duck.com>

* make it work (dirty)

Signed-off-by: pyranota <pyra@duck.com>

* reduce db calls

* remove `triggered_by_relative_import`

Signed-off-by: pyranota <pyra@duck.com>

* add comment to common_dependency_path

Signed-off-by: pyranota <pyra@duck.com>

* add fallback to old behavior

Signed-off-by: pyranota <pyra@duck.com>

* remove TODOs

Signed-off-by: pyranota <pyra@duck.com>

* pass deployed hash to git sync handler function

Signed-off-by: pyranota <pyra@duck.com>

* fix ci

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-08-27 17:44:35 +00:00
Ruben Fiszel
4ad0d255f3 feat: autovacuum or high intensity tables 2025-08-27 17:29:06 +00:00
pyranota
5808840b78 fix(go): could not read Username for 'xyz': terminal prompts disabled (#6478)
* fix(go): could not read Username for 'xyz': terminal prompts disabled

Signed-off-by: pyranota <pyra@duck.com>

* remove unused import

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-08-27 11:11:54 +00:00
pyranota
475f405d06 fix(go): exec: "git": executable file not found (#6475) 2025-08-26 16:07:15 +00:00
hugocasa
8d31c2ab0d feat(backend): support unencrypted connection to mssql (#6453) 2025-08-25 12:39:37 +00:00
Ruben Fiszel
97ed4a539b nits cleanup + faster script index #6450 2025-08-23 22:42:39 +00:00
hugocasa
c13747cda9 fix(frontend): ai agent flow status + UI nits (#6447)
* fix(frontend): ai agent flow status

* nit: prevent undefined node issue

* feat: UI nits + flow status select iter fix

* nit ai agent color in picker
2025-08-22 18:51:13 +01:00
pyranota
d152e8e58f add go1_22_compat annotation (#6432)
* bump go to 1.25

Signed-off-by: pyranota <pyra@duck.com>

* Update Dockerfile

* add comms

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-08-21 15:14:29 +00:00