Commit Graph

946 Commits

Author SHA1 Message Date
Ruben Fiszel
a32a0abd3f handle better empty tag overrides 2024-10-29 09:01:02 +01:00
Ruben Fiszel
94cf0184a8 improve test this step for flow tag overide 2024-10-29 08:19:17 +01:00
Ruben Fiszel
5b7c6d7d62 fix(backend): in flows, workspace scripts should use their set tags instead of the default one 2024-10-29 07:29:39 +01:00
Ruben Fiszel
fe7d044a66 feat: autoscaling v0 (#4593)
* all

* all

all

all

all

all

all

all

all

all

* all

* all

* nits

* nits
2024-10-28 00:47:19 +01:00
Ruben Fiszel
d1a28eb7ca fix: do not update created_at of scripts on lockfile generation 2024-10-25 17:44:41 +02:00
wendrul
5621bd30d3 Make file_resources that are filled with a variable in ansible (#4584)
* Make file_resources that are filled with a variable in ansible

* Rename file_resources section to files

* Add variable example to init code
2024-10-25 14:18:28 +02:00
Stephen Beckstrand
f0eebd9bd7 chore: Update references to windmill-ee-full image (#4578)
* chore: Update windmill-ee-full image references

The messaging in a few backend executors references the need to
use the full enterprise image rather than `windmill-ee`. These
messages reference the `windmill-full-ee` image but its the image
is actually named `windmill-ee-full` under the available Packages.

* chore: Update refs to rust to ansible in ansible executor
2024-10-24 17:40:59 +02:00
Ruben Fiszel
134cfdb30e fix(bash): correctly propagate sigterm for cancelled bash scripts 2024-10-24 14:42:37 +02:00
Ruben Fiszel
248fdc24a6 feat(typescript-bun): support relative imports without the .ts extension 2024-10-18 12:26:57 +02:00
pyranota
b237873a87 Annotations refactor (#4512)
* Refactor annotation system

* Move tests to different location

* Limit annotations per line (15)

* Write initial benchmark

* Optimize

* More optimizations

* Rewrite with 3x speed!

* Remove benchmarks

They were in wrong place and for development

* Polishing

* Remove unused leftover

* Integrate new annotation system

* Add comment explanation

* Remove `hashe-based matching`

Turned to be it is impure and non-deterministic method

* Add `tracing::error` for `// native` without `deno_core`

If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.

This commit prints error to logs

* add more tests

* Update annotations.rs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-17 09:14:31 +02:00
Ruben Fiszel
b7ad19bb75 fix(nsjail): improve memory reading when using nsjail 2024-10-17 07:42:30 +02:00
Ruben Fiszel
1466da3999 fix: add extra args support for exception to bun scripts 2024-10-17 01:19:25 +02:00
Ruben Fiszel
3174024d8e fix: fix script persistence in url + add support for extra error args in python 2024-10-17 01:07:56 +02:00
Ruben Fiszel
a323d2b29a fix deno lockall tests 2024-10-12 17:59:05 +02:00
Ruben Fiszel
42fe31f804 fix: fix deno cache --allow-import on deno 2 2024-10-12 17:18:46 +02:00
Ruben Fiszel
0915968eba allow python multiprocessing on nsjail 2024-10-11 01:40:09 +02:00
Ruben Fiszel
7db74eecb9 add --allow-import to nsjail deno 2024-10-10 15:29:49 +02:00
Ruben Fiszel
b133f19333 nit deno_core 2024-10-10 15:13:55 +02:00
Ruben Fiszel
7ca5bf2fae fix: improve handling of empty lock files on deno 2.0 2024-10-10 04:39:57 +02:00
Ruben Fiszel
0e61f8db78 deno 2.0 use frozen for cache 2024-10-10 04:06:27 +02:00
Ruben Fiszel
c3a5736641 fix: update internal deno runtime to latest (deno 2.0) 2024-10-10 03:42:28 +02:00
Alexander Petric
f831b9b1fe improving pwsh error handling (#4504)
* making pwsh error handling more robust

* cargo fmt
2024-10-09 05:20:46 +02:00
Ruben Fiszel
19c62ba195 fix: fix uv path 2024-10-03 22:42:34 +02:00
Ruben Fiszel
c84e6fd05d fix: flow picker of flows + precache hub scripts as bundles 2024-10-03 20:48:26 +02:00
pyranota
b54c9ee657 feat: Replace pip-compile with uv (#4460)
* Update `shell.nix`

- Replace pip-compile with uv packages
- Pin rust version
- Add var to trigger windmill print more info in stdout

* Replace `pip-compile` with `uv` (dirty + untested)

* Fix arguments passed to uv

Some of the flags are included by default in UV and can be safely removed:
- --resolver=backtracking
- --no-emit-index-url

Also uv does not support `--pip-args` and suggests to directly pass args to uv.

* Remove extra `dbg!`

* Replace 'pip-compile' with 'uv' in Dockerfile

* Add fallback option to `pip-compile` (Disabled)

* Add `uv` to `docker/DockerfileSlim*`

* Add `get_annotation_python` and rename `get_annotation` to `get_annotation_ts`

* Add option to fallback to pip-compile

Put `# no_uv` on top the file for specific python script

Or set `USE_PIP_COMPILE` variable to `true`

* Put back `pip-tools` into shell.nix

* Make sure lockfile resolves again if `#no_uv` used

Add #no_uv to the end of requirements (requirements.in)
That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

* Put `pip install pip-tools` in original spot

* Fix compilation error

* Fix EE compilation error

error[E0658]: attributes on expressions are experimental
   --> windmill-worker/src/python_executor.rs:144:5
    |
144 |     #[cfg(feature = "enterprise")]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information

* Add `no_cache` annotation

Will force recalculation of lockfile
And block uv from using cached values

* Target uv cache to /tmp/windmill/cache

* Prohibit uv from managing python

* Add uv to DockerfileBackendTests

* Pin uv version to 0.4.18 in Dockerfiles

* Dont put `#no_uv` in requirements.in

Instead postfix hash for requirements.in with `-no_uv`

* Push Warning to logs if fallbacked to pip-compile

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 16:00:35 +02:00
Alexander Petric
f5c4727274 feat(worker): support workers to run natively on windows (#4446)
* minimal code change to get windmill worker on windows for bun and python + rustfmt

* adding support for powershell

* compiling error on unix

* rust linting comments

* comments hugo: PSModulePath

* comments ruben, refactor to simplify

* adding build workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* skip migration env, ee fixes

* improvements powershell

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* killing process tree in windows

* sqlx_offline

* install openssl for github windows runner

* used pre-installed openssl

* used pre-installed openssl

* build ee

* build ee

* build ee

* build ee

* adding commented out steps for artifact publishing

* build on tag matchinv v* pattern

* ren instead of mv on Windows

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix merging issue

* gate imports for windows

* fixing default cargo home path...

* fixing default cargo home path...

* comments ruben

* make pwsh default modules loading more robust on unix (#4448)

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 15:59:12 +02:00
Ruben Fiszel
a4a59ef21e deno_core is an optional feature flag (#4473)
* full

* all

* add deno_core as features

* all

* remove warnings

* all
2024-10-03 13:11:58 +02:00
HugoCasa
b70ca6e3ed skip step better UI (#4465) 2024-10-01 12:28:46 +02:00
HugoCasa
0df169e3f9 feat: flow step skipping (#4461) 2024-09-30 16:30:59 +02:00
Ruben Fiszel
fcc51bc5a4 optimize flow update parallel queries 2024-09-29 13:17:11 +02:00
Ruben Fiszel
0df35e5787 add urandom to nsjail 2024-09-29 12:31:55 +02:00
Ruben Fiszel
004bcb2b07 quote req and target for nsjail python download 2024-09-29 12:13:54 +02:00
Ruben Fiszel
8fb3637183 optimize early stop query 2024-09-29 01:19:06 +02:00
Ruben Fiszel
5acfc83521 optimize skip failure query 2024-09-29 00:38:43 +02:00
Ruben Fiszel
30d9c40193 optimize more queries 2024-09-28 17:34:37 +02:00
Ruben Fiszel
363e5e1727 improve suspend_first frequency behavior 2024-09-28 15:50:12 +02:00
Ruben Fiszel
b5e226b977 fix: improve suspend_first behavior and frequency 2024-09-28 15:46:50 +02:00
HugoCasa
0b9ec83036 fix: allow preprocessor to write to args.json on nsjail (#4455) 2024-09-27 21:12:38 +02:00
Ruben Fiszel
cfaea38e92 improve benchmarking tools (#4450) 2024-09-27 10:48:04 +02:00
Ruben Fiszel
9f91b1995a feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
HugoCasa
3ce5587faa feat: add return_last_result annotation to sql (#4443) 2024-09-25 18:11:11 +02:00
wendrul
398a09b741 feat: add static variable and resources support to ansible (#4435)
* Add static variable and resources to ansible

* Add wasm parser to cli

* Fix pulling of resources, rs and playbooks

* Remove unused import, add publish pkg-yaml

* Update package.json

* Update generate-metadata

* Fix null fields being serialized in script metadata

* npm install
2024-09-25 15:05:53 +02:00
HugoCasa
590321fd3c feat: add tag filtering to external JWT authentication (#4425)
* feat: tag filtering jwt ext auth

* move tags to scopes

* fix symlink

* update ee ref
2024-09-25 11:21:19 +02:00
Ruben Fiszel
93fa6a0e97 improve init script error handling 2024-09-24 19:27:00 +02:00
HugoCasa
61f27acbd5 fix: time handling in bun and deno (#4423)
* fix: time handling in bun and deno

* put dates conversion in try catch
2024-09-23 18:00:36 +02:00
wendrul
a500994cc2 fix: various improvements for ansible (#4419)
* Improve error messages, fix dependency job issue

* Add nsjail support for ansible

* Remove invalid comment syntax

* Change nsjail config and ansible controller config to be ephemeral

* Move ansible cfg to later to overwrite possible files named ansible.cfg

* Fix ansible.cfg env var

* Fix tmp dir issue
2024-09-23 16:44:37 +02:00
HugoCasa
304dac3447 feat: http routing (#4339)
* feat: http routing

* all

* feat: improve UI

* final stuff

* fix: sqlx

* fix: nit

* fix: nits

* fix: error handler display

* fix: routes panel perms

* all

* fix: improve ability to paste from macos in vscode extension

* fix lock-write for deno

* all

* cleaning

* fix

* cli preprocessor

* nits

* nits

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-20 15:38:38 +02:00
Ruben Fiszel
69b8754aef fix(bun): never pre-bundle BASE_URL env variable 2024-09-20 09:48:00 +02:00
Ruben Fiszel
623ece8c67 fix(go): put shared directory in job dir and not go parent 2024-09-19 10:05:08 +02:00
wendrul
6855b8da9a feat: ansible playbook support (#4399)
* Ansible execution and parsing

* Working collections and pip dependencies

* Remove unused vars

* File resources logic for ansible

* Make frontend for ansible and resource file

* Format file

* Change naming of file resource and inventory fields

* Add autocomplete for file extension resource creation

* Add endpoint to list file formats for resource types

* Add CLI functionality to pull/push file resources

* Add beta tag to ansible

* Add Full image dockerfiles containing ansible

* Prepare sqlx

* Update cargo.lock

* Update ansible path

* Remove unused imports

* Add back import removed by rust-analyzer

* Improve ansible init code

* Prepare sqlx

* Change dockerfile to make the full windmill image

* Remove old dockerfile

* Improve autocomplete file resource extension select

* Remove comment

* Validate file extension

* Add icons to text file resources

* Remove editability of file resource types

* Remove unused import

* Missing space

* Add yaml parser
2024-09-19 00:21:15 +02:00