Commit Graph

1170 Commits

Author SHA1 Message Date
Diego Imbert
0e316239dd EE Refactor (#5844)
* app compiles with every ee substituted

* Replace all oss files content

* Revert "Replace all oss files content"

This reverts commit ea4017d59f.

* delete all ee

* hide all _ee files under private flag

* hide every oss stuff when private flag set

* pub use *

* gitignore and substitute script

* pub mod for ee needed for ee repo

* small mistakes

* remove oidc_oss impl

* ee ref (temp)

* ee ref

* fix --all-features selecting private in OSS CI

* ee repo ref

* allow unused
2025-06-02 22:12:33 +02:00
pyranota
ed61d97700 feat(rust): shared build directory (#5610)
* feat: rust incremental compilation

* update dockerfiles

* fix compilation error

* add windows flags

* init

* polishing

* update

* return mount-point

* Update backend/windmill-worker/src/rust_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* share worker dir if nsjail off

* final refactor

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-05-29 22:11:35 +02:00
Diego Imbert
fdefd4be93 feat: duckdb sql lang support (#5761) 2025-05-27 15:52:57 +02:00
Ruben Fiszel
5b96bccedd feat: add aws oidc support for instance s3 storage (#5810)
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
2025-05-25 14:03:38 +02:00
pyranota
26222539e6 feat(python): inline script metadata (PEP 723) (#5712)
* make resolver

* more updates

* fix build

* fix raw_dependencies job type

* compat with http agent workers

* refactor

* rename

* more refactor

* cleanup

* more tests

* fix s3

* small fixes

* more fixing

* fix endpoint

* nit: update comment

* update ee ref

* update ee ref

* update ee ref

* implement safer `list_available_python_versions`

* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix build

* update ee ref

* ignore versions <=3.9

* fix windows build

* correct versions filter

* fix windows build (this time for real)

* inject error to debug CI

* update CI

* undo debug of CI

* fix tests

* remove outdated comment

* update ee repo ref

* Update ee-repo-ref.txt

* Update backend/parsers/windmill-parser-py-imports/src/lib.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update InstanceSetting.svelte

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-05-24 02:13:47 +02:00
Ruben Fiszel
af74653b7f improve err message 2025-05-22 19:48:26 +02:00
Ruben Fiszel
7c24fbcef2 fix: improve docker mode unexpected exit handling 2025-05-22 19:37:47 +02:00
Ruben Fiszel
3f825ec77f reduce the pip resolution cache duration 2025-05-22 18:47:02 +02:00
Ruben Fiszel
e3e25daee7 fix 2025-05-22 14:12:10 +02:00
Ruben Fiszel
d662e18f97 add more labels to traces 2025-05-22 12:07:21 +02:00
Ruben Fiszel
dfd8c4cd2a more verbose docker wait errors 2025-05-22 01:07:29 +02:00
HugoCasa
30edcdfe0e fix: workspace preprocessor improvements (#5784)
* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix: workspace preprocessor fixes

* tmp ee ref

* fix build

* update ee ref

* fix: hub script preprocessor handling

* fix build

* good ref

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
2025-05-21 17:49:21 +02:00
pyranota
07c2ff5668 feat(python): add annotation to skip result post-processing (#5769)
* feat(python): add annotation to skip result post-processing

Typically windmill will replace all NaN, Infinity and -Infinity from resulting string.
We do it because JSON specification does not support these types as well as DB.

However it will substitute also the cases when any of those words are used within the string.
E.g. script returning "To Infinity and Beyond" will be postprocessed to "To null and Beyond".
Current behaviour is done for the performance sake and now can be disabled with `#skip_result_postprocessing` annotation.

* add comments

* remove extra comments
2025-05-19 22:31:46 +02:00
Ruben Fiszel
beb658757f nit error 2025-05-17 10:56:16 +02:00
Ruben Fiszel
59afa493fa fix: improve handling of custom concurrency key/tag with preprocessors (#5762)
* fix

* rm spurrious diff

* handle failing preprocessors

* nits

* update

* update
2025-05-17 10:52:28 +02:00
Diego Imbert
b29c6e7636 fix: S3 sql mode returns S3Object (#5764)
* fix upload_s3_file not checking response

* Return S3Object instead of path string
2025-05-16 19:28:57 +02:00
Ruben Fiszel
e87d4f3c1a fix: avoid deadlocks in sending completed job to result processors (#5742)
* buffer

* unbounded and bounded

* fix

* buffer

* impl clone

* flume it all

* update

* update

* update
2025-05-15 09:13:09 +02:00
Diego Imbert
669a95c3be oracle support for s3 streaming + misc (#5738)
* changed SQL templates to include S3

* oracle support for s3 streaming
2025-05-14 15:45:00 +02:00
Ruben Fiszel
5b26498e09 add more duration warnings to logs 2025-05-14 10:13:14 +02:00
Ruben Fiszel
d35a7d22f9 fix: add support for log compaction on docker jobs (#5732)
* improve docker compact

* improve docker compact

* update ref

* update

* update agent workers
2025-05-13 21:42:15 +02:00
wendrul
f73c90c751 fix: Ansible lockfile back compatibility issue (#5731)
* fix: Ansible lockfile back compatibility issue

* Replace condition with more readable alternative
2025-05-13 16:19:18 +02:00
Diego Imbert
c7886ea07a feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate

* S3 works with new syntax

* snowflake s3 streaming support

* postgres s3 support

* fix postgres stream format

* mysql s3 streaming

* mssql s3 streaming

* new s3 mode syntax

* optional folder param

* rename folder to prefix

* json_stream_arr_values

* cargo toml rollback

* convert_ndjson with datafusion

* format conversion kinda works

* Fixed not finishing the datafusion writer

* support for pg and mssql

* fix file ext

* bigquery conversion and works with s3 streaming

* fix s3 flag parser

* snowflake s3 streaming support

* factor out duplicate code

* remove anyhow

* Err case for parse s3 mode

* Send error to mpsc

* bigquery s3 streaming fix for huge queries

* remove extra stuff

* snowflake s3 streaming support

* small regex mistake

* cfg(not(feature = "parquet"))

* fix CI (unused import)

* error handling fix (graphite)
2025-05-13 10:19:44 +02:00
Ruben Fiszel
b51568c166 fix: make ansible more resilient to invalid lockfiles 2025-05-12 11:40:13 +02:00
dieriba
54b4815df2 nits: use generic deserializer for empty data (#5727)
* use generic deserializer for empty data

* update repo ref

* fix import
2025-05-12 11:00:55 +02:00
Ruben Fiszel
a23a4c0faf nit compile 2025-05-11 11:52:42 +02:00
Ruben Fiszel
a3c76fb10c fix: improve error display of nativets exceptions 2025-05-11 11:35:31 +02:00
Ruben Fiszel
7c69959853 fix: improve agents workers handling of WHITELIST_ENVS 2025-05-11 08:38:46 +02:00
HugoCasa
254c3cf8ef feat: preprocessor refactor (#5629)
* feat: preprocessor refactor

* nit

* oneOf kind support + new capture format + fix build

* fix build

* fix build

* update preprocessor templates

* cache runnable version info

* update ee ref

* update hub scripts && ee ref
2025-05-09 09:50:01 +02:00
dieriba
242a565428 feat: raise error if end early in flow (#5653)
* update front

* handle err_msg display if enabled when stop condition is met

* nits

* Update backend/windmill-worker/src/worker_flow.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update openflow.openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update worker_flow.rs

* Update worker_flow.rs

* update flow test

* update condition

* remove println

* nits

* fix flow test

* nits and missing property on struct definition

* nits

* fix unsaved state

* update .sqlx

* cleanup sqlx

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-05-07 18:23:40 +02:00
Ruben Fiszel
9bdd301f52 fix: ansible in agent mode can use inventory.ini 2025-05-06 11:57:08 +02:00
Ruben Fiszel
7eb9d7d46c fix: do not track relative deps for scripts with raw defined deps from CLI (#5696) 2025-05-02 00:37:28 +02:00
Ruben Fiszel
46a3c29c9f improve agent workers 2025-04-29 20:15:49 +02:00
Diego Imbert
35b69da25c fix: skip_flow_update for dependency tracking table (#5670) 2025-04-28 17:57:35 +02:00
Ruben Fiszel
a7b2b51444 fix: improve skip failure on parallel branchall 2025-04-27 00:37:25 +02:00
wendrul
fdd1642ce1 feat: Ansible improvements (vault, roles and git repos) (#5655)
* Ansible vault + roles

* Clone arbitrary repos

* Fix cloning logic after merge

* Make function for cloning without history any commit

* Cloning repos and lockfile on the commit

* Improve error messages

* Create lockfile for roles and collections

* Simplify ansible ssh identity interface

* Ansible vault password: pass just a variable instead of 2 step approach

* Lock lockfiles for roles and collections

* fix typo

* Change git ssh identity section name

* Rename variable

* Update init script for ansible

* Suppress error when no roles

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-04-24 16:04:28 +02:00
Ruben Fiszel
e316dbd9bd fix: track relative imports in python and ts even if lockfile is provided 2025-04-23 08:40:22 +02:00
Diego Imbert
5bca8f60e9 fix: improve MySQL datetime parser timezone handling (WIN-1155) (#5645)
* fix mysql datetime parser for non tz dates

* static regexes
2025-04-21 14:46:28 +02:00
Ruben Fiszel
3c68fef2ab ui code builder v0 (at secret path) (#4964) 2025-04-20 01:19:02 +02:00
Ruben Fiszel
87dbb14656 improve no_proxy value by default 2025-04-17 00:05:47 +02:00
Ruben Fiszel
24e893b8c5 fix: flow editor workspace script test use actual workspace script hash 2025-04-16 00:44:49 +00:00
Ruben Fiszel
496960d349 improve last_job_suspended_history 2025-04-12 15:02:43 +00:00
Ruben Fiszel
8c6e620f5c nit benchmarks 2025-04-12 12:54:05 +00:00
Ruben Fiszel
12ba15c928 improve benchmarks 2025-04-12 11:09:21 +00:00
pyranota
0b6d017fed feat(python): per import requirement pin (#5520)
* implement single line pin

* make panic-safe

* use pin even if multiple modules imported withing single statement

* add repins and make imports respect pins

* keep all pins

* Allow multiple pins

* add comments + handle stuff more safely

* fix fully qualified imports

* remove ignore

* sort nested

* apply unique to output requirements list

* fix typo

* remove mut

* update sqlx

* sort imports

* sort imports

* fix formatter and format

* refactor

* fix comptime error

* write tests

* perf: do not capture if string is empty
2025-04-11 21:31:51 +00:00
Ruben Fiszel
d5186da271 allow multiple workers on agent mode (#5607) 2025-04-11 21:19:29 +00:00
HugoCasa
b9e879618b feat: signed s3 objects (#5593)
* feat: accept signed s3 objects for s3 file keys in apps + sign endpoint and helpers

* Update backend/windmill-api/openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* Update python-client/wmill/wmill/client.py

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nti

* fix build

* fix build

* presigned

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* b

* fix sqlx

* nit

* typo

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-04-11 18:03:40 +02:00
dieriba
6339775404 feat: add gcp trigger (#5501)
* update: add migration for gcp_trigger table, add cli method gcp and add gcp_trigger args for github ci

* update: add gcp module and routing it in lib.rs

* update: added gcp type to TRIGGER KIND type in db, add new feature condition to try_get_fn and added gcp_trigger feature in cargo file

* update: add ui for gcp trigger, added missing triggers type for job kin, add gcp trigger in deploy to functionallity

* adding google cloud crate

* update: handle pull and push delivery

* update: handle pull and push delivery

* update: handle push event, changed front ui

* update: ui

* fix: capture for gcp

* update: automatically manage pub sub subscription and refactoring

* capture done

* update cli types

* fix: wrong func argument and type openapi

* fix: missing import

* update .sqlx

* update: svg color and remove pulse button for capture push gcp

* update: handle deployto

* update script helper and link to hub gcp script/flow

* update gcp representation

* update: add confirmation modal

* add unique index to mitigate same subscriber id, fix `zombie worker`, update test connection logic

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update cli/gen/services.gen.ts

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* nits

* update repo ref

* fix feature function

* update dependencies

* update .sqlx

* fix missing import

* update: handle existing subscription and creating new one or update it

* fix ee build

* handle existing config properly

* update .sqlx

* update .sqlx

* remove unused

* update documentation link fix ci

* use on instance of empty_string_as_none fn

* update refo ref

* updat script_helper

* update ui

* update migration

* add missing arguments

* fix and nits

* update repo ref

* remove unused

* nits

* add doc links

* nits

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerPanel.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* use trigger path as default route path

* update .sqlx

* update repo ref

* update ref

* update route path

* update types

* fix deploy to

* nits

* update repo ref

* update .sqlx

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorConfigSection.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* nits

* nits

* nits

* remove dupliacte fn and update repo ref

* remove unused import

* fix missing import

* fix: wong name var

* update script helper and template script

* Update frontend/src/lib/script_helpers.ts

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update ee-repo-ref.txt

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-04-10 20:39:01 +00:00
Ruben Fiszel
1f1164199b nit 2025-04-10 01:00:29 +00:00
Ruben Fiszel
63fa499015 feat: agent workers v2 using http (#5588) 2025-04-10 00:56:11 +00:00
HugoCasa
b4f8c88c19 fix: mssql ca_cert deserializing (#5587) 2025-04-08 17:12:20 +02:00