Commit Graph

1111 Commits

Author SHA1 Message Date
Ruben Fiszel
d4f61f13fd fix(bash): allow process substitution on nsjail 2025-02-18 16:46:42 +01:00
Ruben Fiszel
138cedf1da fix(bash): improve bash last line as result reliability using bash process substitution (#5321) 2025-02-18 15:13:28 +01:00
Ruben Fiszel
953082681e fix: improve que job indices for faster performances 2025-02-17 21:26:46 +01:00
HugoCasa
28558e674f feat: parse script for preprocessor/no_main_func on deploy (#5292)
* feat: parse script for preprocessor/no_main_func on deploy

* fix substitute for linux
2025-02-14 15:42:18 +01:00
Ruben Fiszel
77a8eedc96 fix: display branch chosen even if emoty branch 2025-02-14 04:23:25 +01:00
Ruben Fiszel
e7d4582789 fix: improve handling of empty branches and loops 2025-02-14 04:12:55 +01:00
Ruben Fiszel
fe922114a7 fix(bun): remove unecessary buntar in a bun bundle world 2025-02-13 10:27:48 +01:00
Ruben Fiszel
2015e79ff0 fix: better handling of null pre-processor return values 2025-02-12 22:10:53 +01:00
Ruben Fiszel
fcbfef4c31 fix encrypted values passed from apps in flows 2025-02-12 20:07:27 +01:00
Lucas Abel
bf206515e8 fix(backend): improve schedule queries plan to leverage indices better for performance (#5273)
* backend: fix schedule queries plan

* rework

* fix skipped as non success

* update sqlx

* update

* all

* fix npm run check

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-02-12 16:34:03 +01:00
wendrul
6357ed3d5e fix: Remove cache dir mount and mount only the cache executable (Rust, C#) (#5270) 2025-02-11 16:08:19 +01:00
Lucas Abel
bb951a4434 backend: fix missing virtual modules when inserting flow version lite (#5257) 2025-02-10 18:55:31 +01:00
Lucas Abel
4a744ff313 backend: move bench.rs from common to worker crate (#5253) 2025-02-10 11:39:59 +01:00
Ruben Fiszel
bf7f67e42f load previous flow steps even for just test this steps 2025-02-10 04:38:29 +01:00
Ruben Fiszel
403826fca9 fix: worker name in job + better timeout handling for same_worker jobs (#5248) 2025-02-08 22:20:06 +01:00
Lucas Abel
4849af04de backend: fix workflow as code (#5239)
* backend: improve `/get_job_update` after v2

* backend: insert missing `workflow_as_code_status` on completion

also insert `flow_status` from so we can remove the query when `_duration` is above 500

* backend: fix workflow_as_code after v2

* backend: add `workflow_as_code` worker test
2025-02-07 12:27:43 +01:00
HugoCasa
3bb19dd87c fix: remove unnecessary rw on cache for powershell in nsjail (#5236) 2025-02-06 18:49:21 +01:00
Ruben Fiszel
de1e1563ed remove go cache 2025-02-06 17:41:47 +01:00
Lucas Abel
d51c3080f7 backend: use v2 tables through views where possible (v2 phase 3) (#5119) 2025-02-06 12:43:24 +01:00
Lucas Abel
e0d99ab703 backend: check queries to queue and completed_job where possible (v2 phase 2) (#5125) 2025-02-06 12:39:32 +01:00
Lucas Abel
3e8e0c653e backend: migrate job tables to v2 schema (v2 phase 1) (#5084) 2025-02-06 12:37:11 +01:00
Ruben Fiszel
e1f885b206 delete proto file after install 2025-02-06 00:49:45 +01:00
Ruben Fiszel
abf094bcbc fix uv install for nsjail (#5222) 2025-02-06 00:00:33 +01:00
Ruben Fiszel
6d33cd24e9 remove rw access to cache dir in php 2025-02-05 22:17:27 +01:00
Ruben Fiszel
1769d62c96 removing rw access to cache dir in uv pip install 2025-02-05 22:08:38 +01:00
Ruben Fiszel
f8cc732ce0 removing rw access to cache dir in uv pip install 2025-02-05 22:06:52 +01:00
Ruben Fiszel
3c6d1029d1 remove rw access to bun cache in nsjail mode 2025-02-05 21:45:37 +01:00
Ruben Fiszel
e7e751c3bf improve debuggability of failed native jobs 2025-02-05 18:35:09 +01:00
Ruben Fiszel
645be25bef nit 2025-02-05 17:15:38 +01:00
Ruben Fiszel
90ba65ae20 improve error messages for internal err 2025-02-05 16:32:01 +01:00
Ruben Fiszel
809242987e improve errors II 2025-02-05 16:18:20 +01:00
Ruben Fiszel
32b75b8d84 improve error messages with locations 2025-02-05 15:56:58 +01:00
HugoCasa
8a446a658a fix: preprocessor args python (#5210) 2025-02-04 23:34:03 +01:00
pyranota
97c11340c3 fix(python): clear env before installing/finding python (#5209)
* fix(python): clear env before installing/finding python

* add windows-specific variables
2025-02-04 18:58:22 +01:00
pyranota
0e80775d6d feat(python): make S3 cache arch specific (#5196)
* add .minio-data to .gitignore

* feat(python): make S3 cache arch specific

Current schema:

S3-Bucket
├── python_311
│   ├── wheel==1.0
│   └── wheel2==1.0
└── python_312
    ├── wheel==1.0
    └── wheel2==1.0

New schema:

S3-Bucket
├── linux_aarch64
│   └── ...
└── linux_x86_64
    ├── python_311
    │   ├── wheel==1.0
    │   └── wheel2==1.0
    └── python_312
        ├── wheel==1.0
        └── wheel2==1.0

* remove .minio-data from .gitignore

* remove unneeded tracing::error
2025-02-03 19:31:55 +01:00
Ruben Fiszel
0c4901f802 nit fix hub cache init on server 2025-02-01 08:55:44 +01:00
Ruben Fiszel
dc00a13501 use sync decrypt 2025-01-31 10:31:19 +01:00
Alexander Petric
03ad0385ad improve audit aprovals (#5161)
* update audit log to track aproval and cancellation of job

* approval to resource field

* audit resource as json

* refactor aproval audit logs

* moving audit logs to worker_flow

* striping u/ from auditor name

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-30 09:13:44 +01:00
pyranota
c60f8dacf1 feat(python): add custom_wheels directory to PYTHONPATH (#5169)
* fix(python): fix uv can't find ssl certificates

- Add `PY_NATIVE_CERT` flag, forces UV to use native tls
- Rename `PIP_INDEX_CERT` to `PY_INDEX_CERT`
- Rename `PIP_TRUSTED_HOST` to `PY_TRUSTED_HOST`

For backwards compatibility PIP* variables are still accessible

* feat(python): add `custom_wheels` directory to PYTHONPATH

Add global directory by path `<CACHE_DIR>/python_xyz/custom_wheels`

For example for scripts running python 3.11, in every execution
`<CACHE_DIR>/python_311/custom_wheels` will be accessible and all wheels placed there could be imported and used.

This is usefull for preinstalling wheels before runtime

* Make it work with Nsjail

* Rework and make custom_wheels optional

* Remove `create_dir_all` from imports

* Use sync version of metadata

* Rename `custom_wheels` to `global-site-packages`
2025-01-29 19:13:47 +01:00
pyranota
680fb1852a fix(python): fix uv can't find ssl certificates (#5157)
- Add `PY_NATIVE_CERT` flag, forces UV to use native tls
- Rename `PIP_INDEX_CERT` to `PY_INDEX_CERT`
- Rename `PIP_TRUSTED_HOST` to `PY_TRUSTED_HOST`

For backwards compatibility PIP* variables are still accessible
2025-01-29 15:07:00 +01:00
Alexander Petric
fdb712234e feat: mssql executor supports azure db redirect (#5150)
* feat: mssql executor supports azure db redirect

* notes

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-27 22:45:31 +01:00
Ruben Fiszel
e3d6aef109 fix compile 2025-01-27 22:00:48 +01:00
Ruben Fiszel
2800903c9a change windmill-ee-full error message 2025-01-27 15:47:23 +01:00
Ruben Fiszel
da918bee83 change windmill-ee-full error message 2025-01-27 15:41:43 +01:00
pyranota
2f1804b6d0 fix: Fix python refetching wheels from S3 (#5133) 2025-01-25 12:38:32 +01:00
pyranota
74bb660b97 fix(bun, windows): Fix "A required privilege is not held by the client." (#5126) 2025-01-24 15:32:29 +01:00
dieriba
316a2167e7 feat: postgres triggers (#4860)
* feat: init database triggers

* feat:  wip: database_triggers

* feat:  add database triggers front view

* feat: 🚧 database_triggers

* feat: 🚧 add definition in yaml, updated backend code and added migration

* feat: 🚧 updated migration file, update openapi.yml, updated database_triggers page and backend function

* fix: struct rust

* feat: 🚧 update migrate, database trigger backend function fixed

* feat: 🚧 add resource picker front, update backend function

* feat: 🚧 edit inner database inner, update triggers

* feat: 🚧 database_triggers

* feat: 🚧 update openapi yaml, prettied websocker trigger

* feat: 🚧 database_triggers

* feat: 🚧 add resource module, update variable file, working on main loop for database_trigger

* feat: 🐛 working sqlx query

* feat: 🚧 fix query with sqlx, added main loop

* feat: 🚧 add new column database_trigg

* feat: 🛂 run jobs works

* feat: 🚧 handling slot name and replication

* feat: restructring triggers, decoding trigger message on work

* feat: 🚧 database_trigger

* feat: 🚧

* feat: 🚧 converter done, work on custom script

* feat: 🚧 multiple trigger

* feat: 🚧 adding new argument function

* feat: 🚧 database_trigger

* feat: 🚧 add generate template for front, update script picker

* feat: 🚧 template script fix bug, work on restructing backend logic

* feat: 🚧 update autogenerated script, add persistence state for template script

* feat: 🚧 update structure client

* feat: 🚧 rewrited crud function

* feat: 🚧 added publication handler

* feat: 🚧 new ui finished

* feat: 🚧 added slot function hanlder finish front ux

* feat:  ux improvement done, backend logic done

* feat: 🐛 fix where clause

* feat:

* feat:

* chore: update .sqlx and remove empty package

* fix: update publication and remove unneccessary print

* feat: finish converter to json, remove save button, remove unused crate

* feat: update migration for database trigger, fixed query, fixed frontend error with missing feature on back

* chore: update .sqlx

* chore: update sqlx

* chore: .sqlx

* chore: add unused

* fix: use right database resource in back and front

* fix build

* refactor:

* fix sqlx

* nits: retrieve all script from database trigger editor

* fix: template script object, fix route name

* merge main

* feat: rework ux

* feat/finish ux and update backend logic

* feat: database_trigger

* feat: database_trigger

* chore: update sqlx

* chore: fix ci

* nits: fix worker.rs

* feat: database_trigger

* Update frontend/src/lib/components/triggers/database/DatabaseTriggersPanel.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* feat:

* fix ci

* nits: database to postgres

* nits: fix .sqlx

* feat: add link to original code and open tab functionallity

* nits: update .sqlx

* nits: fix request type and yaml file

* nits: add abel suggestion

* fix: update .sqlx

* fix: update .sqlx

* nits: add comment to hex.rs

* nits: remove features, rename database to postgres name

* nits: replace database name by postgres

* nits: database name to postgres

* nits: update .sqlx

* nits: update .sqlx

* nit

* nit

* nits

* nit

* nits

* nits

* nits

* nits

* fix sqlx

* nits: update .vscode

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-24 15:12:41 +01:00
pyranota
e47dd697f9 feat(python): Multiple runtime versions (#4579)
* feat: Handle `pip install` by `uv`

Dirty and untested, but already something working

* Integrate with NSJAIL and prepare fallbacks

* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific

* Remove `--disable-pip-version-check`
Reason:
   warning: pip's `--disable-pip-version-check` has no effect

* Fix backend compilation error

* Pip fallback overwrite UV's cache

* Initially refactor cache (No S3)

* Support S3

* Remove unused import

* Handle flags for NSJAIL

* Return deleted flag

* Remove verbose mode and enable link-mode=copy

* Granural migration of lockfiles

Before i realized we dont need it :)

* Initial draft (not-working)

* Add fallback

* Fix bug preventing uv from installing deps

'\n' - Love it

* Add verbosity indicator

* Iterate on feature
- Added instance python version
- Rework logic

* Fix EE build error
error[E0599]: no method named `iter` found for tuple `(PyVersion, std::vec::Vec<std::string::String>)` in the current scope

* Support S3

* Support NSJAIL

* Refactor `get_python`

* Make NSJAIL work [Unsafe]

config file missed /proc mount causing install phase to fail

* Trigger CI

* Clean up

* Make Actions build it

* Trigger CI #2

* Update Dockerfile and clean up

* Change fallbacks
now there is only no_uv and NOUV

* Expose INSTANCE_PYTHON_VERSION through env variable

* Change namings

* Include py-version to requirements.in

Also add comments and make code much cleaner

* Use const for python installation dir

It was hardcoded before

* Pin preinstalled version

* Update python_executor.rs

* Up to date branch

* Create PYCACHE dirs

TODO: PY_TAR_DIRS

* Fix after merge

* Make it safer

* Implement USE_SYSTEM_PYTHON

* Implement latest_stable option

* Load INSTANCE_PYTHON_VERSION on startup

* Check for multiple annotations used

* Fix Latest Stable button not pressed if selected

* Proper error handling for conflict on multiple annotations

* Fix merge conflicts

* Preinstall 3.11 and Latest Stable

* Preinstall latest stable in non-blocking manner

* Fix Warning

* Gate preinstall logic behind "python" feature

* Handle raw_deps properly

* Make it work with nsjail

* Revert docker-image.yml

* Revert Dockerfile

* Cleanup + Fixing

* Add windows support

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-24 11:42:48 +01:00
pyranota
fcfad69195 feat: Migrate to bun.lock (#5112)
* bun: Migrate to bun.lock

(In backwards compatible way)

Read more: https://bun.sh/blog/bun-lock-text-lockfile

* Clean up

* More clean up

* Mount bun.lock to jailed process
2025-01-22 12:32:17 +01:00
pyranota
8eb5e72ccd fix(python): Replace Inf with null (#5099) 2025-01-20 19:38:35 +01:00