Commit Graph

1168 Commits

Author SHA1 Message Date
Diego Imbert
d5e9213fd5 feat: duckdb sql lang support (#5761) 2025-05-27 15:52:57 +02:00
Ruben Fiszel
e15b94692d feat: add aws oidc support for instance s3 storage (#5810)
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
2025-05-25 14:03:38 +02:00
pyranota
2833d6da3f feat(python): inline script metadata (PEP 723) (#5712)
* make resolver

* more updates

* fix build

* fix raw_dependencies job type

* compat with http agent workers

* refactor

* rename

* more refactor

* cleanup

* more tests

* fix s3

* small fixes

* more fixing

* fix endpoint

* nit: update comment

* update ee ref

* update ee ref

* update ee ref

* implement safer `list_available_python_versions`

* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix build

* update ee ref

* ignore versions <=3.9

* fix windows build

* correct versions filter

* fix windows build (this time for real)

* inject error to debug CI

* update CI

* undo debug of CI

* fix tests

* remove outdated comment

* update ee repo ref

* Update ee-repo-ref.txt

* Update backend/parsers/windmill-parser-py-imports/src/lib.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update InstanceSetting.svelte

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-05-24 02:13:47 +02:00
Ruben Fiszel
d7d67f3e62 improve err message 2025-05-22 19:48:26 +02:00
Ruben Fiszel
81dfc0abdb fix: improve docker mode unexpected exit handling 2025-05-22 19:37:47 +02:00
Ruben Fiszel
ab632ba92a reduce the pip resolution cache duration 2025-05-22 18:47:02 +02:00
Ruben Fiszel
a268bb87e4 fix 2025-05-22 14:12:10 +02:00
Ruben Fiszel
729d90f61e add more labels to traces 2025-05-22 12:07:21 +02:00
Ruben Fiszel
32b473c6ad more verbose docker wait errors 2025-05-22 01:07:29 +02:00
HugoCasa
23cce4ddb0 fix: workspace preprocessor improvements (#5784)
* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix: workspace preprocessor fixes

* tmp ee ref

* fix build

* update ee ref

* fix: hub script preprocessor handling

* fix build

* good ref

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
2025-05-21 17:49:21 +02:00
pyranota
14fd6458a8 feat(python): add annotation to skip result post-processing (#5769)
* feat(python): add annotation to skip result post-processing

Typically windmill will replace all NaN, Infinity and -Infinity from resulting string.
We do it because JSON specification does not support these types as well as DB.

However it will substitute also the cases when any of those words are used within the string.
E.g. script returning "To Infinity and Beyond" will be postprocessed to "To null and Beyond".
Current behaviour is done for the performance sake and now can be disabled with `#skip_result_postprocessing` annotation.

* add comments

* remove extra comments
2025-05-19 22:31:46 +02:00
Ruben Fiszel
985ef9cd9a nit error 2025-05-17 10:56:16 +02:00
Ruben Fiszel
a499f9dffe fix: improve handling of custom concurrency key/tag with preprocessors (#5762)
* fix

* rm spurrious diff

* handle failing preprocessors

* nits

* update

* update
2025-05-17 10:52:28 +02:00
Diego Imbert
e88954a217 fix: S3 sql mode returns S3Object (#5764)
* fix upload_s3_file not checking response

* Return S3Object instead of path string
2025-05-16 19:28:57 +02:00
Ruben Fiszel
123657fa63 fix: avoid deadlocks in sending completed job to result processors (#5742)
* buffer

* unbounded and bounded

* fix

* buffer

* impl clone

* flume it all

* update

* update

* update
2025-05-15 09:13:09 +02:00
Diego Imbert
d9916318b1 oracle support for s3 streaming + misc (#5738)
* changed SQL templates to include S3

* oracle support for s3 streaming
2025-05-14 15:45:00 +02:00
Ruben Fiszel
ef068ab5ef add more duration warnings to logs 2025-05-14 10:13:14 +02:00
Ruben Fiszel
95cf9c6814 fix: add support for log compaction on docker jobs (#5732)
* improve docker compact

* improve docker compact

* update ref

* update

* update agent workers
2025-05-13 21:42:15 +02:00
wendrul
4662cfd67d fix: Ansible lockfile back compatibility issue (#5731)
* fix: Ansible lockfile back compatibility issue

* Replace condition with more readable alternative
2025-05-13 16:19:18 +02:00
Diego Imbert
25bbdba1d1 feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate

* S3 works with new syntax

* snowflake s3 streaming support

* postgres s3 support

* fix postgres stream format

* mysql s3 streaming

* mssql s3 streaming

* new s3 mode syntax

* optional folder param

* rename folder to prefix

* json_stream_arr_values

* cargo toml rollback

* convert_ndjson with datafusion

* format conversion kinda works

* Fixed not finishing the datafusion writer

* support for pg and mssql

* fix file ext

* bigquery conversion and works with s3 streaming

* fix s3 flag parser

* snowflake s3 streaming support

* factor out duplicate code

* remove anyhow

* Err case for parse s3 mode

* Send error to mpsc

* bigquery s3 streaming fix for huge queries

* remove extra stuff

* snowflake s3 streaming support

* small regex mistake

* cfg(not(feature = "parquet"))

* fix CI (unused import)

* error handling fix (graphite)
2025-05-13 10:19:44 +02:00
Ruben Fiszel
c31aedabab fix: make ansible more resilient to invalid lockfiles 2025-05-12 11:40:13 +02:00
dieriba
00faf1eefe nits: use generic deserializer for empty data (#5727)
* use generic deserializer for empty data

* update repo ref

* fix import
2025-05-12 11:00:55 +02:00
Ruben Fiszel
4bf2c38cbf nit compile 2025-05-11 11:52:42 +02:00
Ruben Fiszel
c76bc4a9cc fix: improve error display of nativets exceptions 2025-05-11 11:35:31 +02:00
Ruben Fiszel
5061358c89 fix: improve agents workers handling of WHITELIST_ENVS 2025-05-11 08:38:46 +02:00
HugoCasa
d2787cfd78 feat: preprocessor refactor (#5629)
* feat: preprocessor refactor

* nit

* oneOf kind support + new capture format + fix build

* fix build

* fix build

* update preprocessor templates

* cache runnable version info

* update ee ref

* update hub scripts && ee ref
2025-05-09 09:50:01 +02:00
dieriba
b2c7bdff13 feat: raise error if end early in flow (#5653)
* update front

* handle err_msg display if enabled when stop condition is met

* nits

* Update backend/windmill-worker/src/worker_flow.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update openflow.openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update worker_flow.rs

* Update worker_flow.rs

* update flow test

* update condition

* remove println

* nits

* fix flow test

* nits and missing property on struct definition

* nits

* fix unsaved state

* update .sqlx

* cleanup sqlx

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-05-07 18:23:40 +02:00
Ruben Fiszel
46b8ac93ae fix: ansible in agent mode can use inventory.ini 2025-05-06 11:57:08 +02:00
Ruben Fiszel
ed6313258b fix: do not track relative deps for scripts with raw defined deps from CLI (#5696) 2025-05-02 00:37:28 +02:00
Ruben Fiszel
0e0c9dd579 improve agent workers 2025-04-29 20:15:49 +02:00
Diego Imbert
e1ead53ce5 fix: skip_flow_update for dependency tracking table (#5670) 2025-04-28 17:57:35 +02:00
Ruben Fiszel
33380a8cd9 fix: improve skip failure on parallel branchall 2025-04-27 00:37:25 +02:00
wendrul
cfe05fa1b8 feat: Ansible improvements (vault, roles and git repos) (#5655)
* Ansible vault + roles

* Clone arbitrary repos

* Fix cloning logic after merge

* Make function for cloning without history any commit

* Cloning repos and lockfile on the commit

* Improve error messages

* Create lockfile for roles and collections

* Simplify ansible ssh identity interface

* Ansible vault password: pass just a variable instead of 2 step approach

* Lock lockfiles for roles and collections

* fix typo

* Change git ssh identity section name

* Rename variable

* Update init script for ansible

* Suppress error when no roles

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-04-24 16:04:28 +02:00
Ruben Fiszel
77bd5b2458 fix: track relative imports in python and ts even if lockfile is provided 2025-04-23 08:40:22 +02:00
Diego Imbert
b8a328a870 fix: improve MySQL datetime parser timezone handling (WIN-1155) (#5645)
* fix mysql datetime parser for non tz dates

* static regexes
2025-04-21 14:46:28 +02:00
Ruben Fiszel
031e522a6a ui code builder v0 (at secret path) (#4964) 2025-04-20 01:19:02 +02:00
Ruben Fiszel
55063e28c7 improve no_proxy value by default 2025-04-17 00:05:47 +02:00
Ruben Fiszel
c4ceb51bdb fix: flow editor workspace script test use actual workspace script hash 2025-04-16 00:44:49 +00:00
Ruben Fiszel
f79ce8b5bf improve last_job_suspended_history 2025-04-12 15:02:43 +00:00
Ruben Fiszel
4f6ef903d0 nit benchmarks 2025-04-12 12:54:05 +00:00
Ruben Fiszel
216b5b01e3 improve benchmarks 2025-04-12 11:09:21 +00:00
pyranota
6fc87dd19d feat(python): per import requirement pin (#5520)
* implement single line pin

* make panic-safe

* use pin even if multiple modules imported withing single statement

* add repins and make imports respect pins

* keep all pins

* Allow multiple pins

* add comments + handle stuff more safely

* fix fully qualified imports

* remove ignore

* sort nested

* apply unique to output requirements list

* fix typo

* remove mut

* update sqlx

* sort imports

* sort imports

* fix formatter and format

* refactor

* fix comptime error

* write tests

* perf: do not capture if string is empty
2025-04-11 21:31:51 +00:00
Ruben Fiszel
d7e215798b allow multiple workers on agent mode (#5607) 2025-04-11 21:19:29 +00:00
HugoCasa
b39209ad3d feat: signed s3 objects (#5593)
* feat: accept signed s3 objects for s3 file keys in apps + sign endpoint and helpers

* Update backend/windmill-api/openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* Update python-client/wmill/wmill/client.py

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nti

* fix build

* fix build

* presigned

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* b

* fix sqlx

* nit

* typo

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-04-11 18:03:40 +02:00
dieriba
06586596e5 feat: add gcp trigger (#5501)
* update: add migration for gcp_trigger table, add cli method gcp and add gcp_trigger args for github ci

* update: add gcp module and routing it in lib.rs

* update: added gcp type to TRIGGER KIND type in db, add new feature condition to try_get_fn and added gcp_trigger feature in cargo file

* update: add ui for gcp trigger, added missing triggers type for job kin, add gcp trigger in deploy to functionallity

* adding google cloud crate

* update: handle pull and push delivery

* update: handle pull and push delivery

* update: handle push event, changed front ui

* update: ui

* fix: capture for gcp

* update: automatically manage pub sub subscription and refactoring

* capture done

* update cli types

* fix: wrong func argument and type openapi

* fix: missing import

* update .sqlx

* update: svg color and remove pulse button for capture push gcp

* update: handle deployto

* update script helper and link to hub gcp script/flow

* update gcp representation

* update: add confirmation modal

* add unique index to mitigate same subscriber id, fix `zombie worker`, update test connection logic

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update cli/gen/services.gen.ts

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* nits

* update repo ref

* fix feature function

* update dependencies

* update .sqlx

* fix missing import

* update: handle existing subscription and creating new one or update it

* fix ee build

* handle existing config properly

* update .sqlx

* update .sqlx

* remove unused

* update documentation link fix ci

* use on instance of empty_string_as_none fn

* update refo ref

* updat script_helper

* update ui

* update migration

* add missing arguments

* fix and nits

* update repo ref

* remove unused

* nits

* add doc links

* nits

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerPanel.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* use trigger path as default route path

* update .sqlx

* update repo ref

* update ref

* update route path

* update types

* fix deploy to

* nits

* update repo ref

* update .sqlx

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorConfigSection.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* nits

* nits

* nits

* remove dupliacte fn and update repo ref

* remove unused import

* fix missing import

* fix: wong name var

* update script helper and template script

* Update frontend/src/lib/script_helpers.ts

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update ee-repo-ref.txt

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-04-10 20:39:01 +00:00
Ruben Fiszel
a7d916b3aa nit 2025-04-10 01:00:29 +00:00
Ruben Fiszel
947c91809d feat: agent workers v2 using http (#5588) 2025-04-10 00:56:11 +00:00
HugoCasa
02e2cf6599 fix: mssql ca_cert deserializing (#5587) 2025-04-08 17:12:20 +02:00
Piyush मिश्र
bdf8cbc4e2 function takes 13 arguments but 14 arguments were supplied (#5577) 2025-04-06 20:38:39 +00:00
Piyush मिश्र
2066451a5b fix: Dynamic select does not work with tag //native (#5576)
closes #5490
2025-04-05 20:43:53 +00:00