Commit Graph

130 Commits

Author SHA1 Message Date
pyranota
deff3b0045 Change TMP for windows 2024-11-14 11:39:52 +00:00
Alexander Petric
d598731913 fixing for windows 2024-11-13 17:07:14 -05:00
pyranota
5a96d64183 Merge branch 'main' into python-uv 2024-11-13 18:39:10 +00:00
Ruben Fiszel
50861fccc0 fix: support multiple pip-extra-index-url with commas 2024-11-12 23:31:00 +01:00
Ruben Fiszel
eff3e1c43a Update python_executor.rs 2024-11-03 02:57:00 +01:00
Ruben Fiszel
844acbf00b Merge branch 'main' into python-uv 2024-11-02 19:48:30 +01:00
Ruben Fiszel
e11cc13c92 fix: apply NO_PROXY and HTTP_PROXY, HTTPS_PROXY more consistently 2024-11-02 13:57:30 +01:00
pyranota
e0040fe3b3 Merge branch 'main' into python-uv 2024-10-31 14:03:32 +00:00
pyranota
ef37928865 Add rollback for multipython (#4610) 2024-10-31 14:27:34 +01:00
pyranota
c839bae160 Fix flock and windows 2024-10-29 17:51:18 +00:00
pyranota
af70339d07 Fix NSJAIL INDEX_URL 2024-10-29 17:34:52 +00:00
pyranota
b1a4f2ab50 Add --link-mode=copy and remove -v 2024-10-23 12:04:27 +00:00
pyranota
59e8004f99 Return deleted flag 2024-10-22 16:57:15 +00:00
pyranota
1271a65b5a Handle flags for NSJAIL 2024-10-22 16:00:43 +00:00
pyranota
53909750ef Support S3 2024-10-22 15:47:47 +00:00
pyranota
60e7fa764c Initially refactor cache (No S3) 2024-10-22 15:31:56 +00:00
pyranota
a0c86ef4c7 Pip fallback overwrite UV's cache 2024-10-22 14:27:32 +00:00
pyranota
bd7607d162 Remove --disable-pip-version-check
Reason:
   warning: pip's `--disable-pip-version-check` has no effect
2024-10-21 18:23:39 +00:00
pyranota
7985a7bafa Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific
2024-10-21 18:16:10 +00:00
pyranota
13896a5741 Integrate with NSJAIL and prepare fallbacks 2024-10-21 18:08:35 +00:00
pyranota
e881ed5e00 Merge branch 'main' into python-uv 2024-10-17 13:18:58 +00:00
pyranota
c3ba832444 Annotations refactor (#4512)
* Refactor annotation system

* Move tests to different location

* Limit annotations per line (15)

* Write initial benchmark

* Optimize

* More optimizations

* Rewrite with 3x speed!

* Remove benchmarks

They were in wrong place and for development

* Polishing

* Remove unused leftover

* Integrate new annotation system

* Add comment explanation

* Remove `hashe-based matching`

Turned to be it is impure and non-deterministic method

* Add `tracing::error` for `// native` without `deno_core`

If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.

This commit prints error to logs

* add more tests

* Update annotations.rs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-17 09:14:31 +02:00
Ruben Fiszel
fac41fd700 fix: fix script persistence in url + add support for extra error args in python 2024-10-17 01:07:56 +02:00
pyranota
d6a654aaff feat: Handle pip install by uv
Dirty and untested, but already something working
2024-10-10 12:51:57 +00:00
Ruben Fiszel
7c0950d9da fix: fix uv path 2024-10-03 22:42:34 +02:00
pyranota
1e7a37ac8b feat: Replace pip-compile with uv (#4460)
* Update `shell.nix`

- Replace pip-compile with uv packages
- Pin rust version
- Add var to trigger windmill print more info in stdout

* Replace `pip-compile` with `uv` (dirty + untested)

* Fix arguments passed to uv

Some of the flags are included by default in UV and can be safely removed:
- --resolver=backtracking
- --no-emit-index-url

Also uv does not support `--pip-args` and suggests to directly pass args to uv.

* Remove extra `dbg!`

* Replace 'pip-compile' with 'uv' in Dockerfile

* Add fallback option to `pip-compile` (Disabled)

* Add `uv` to `docker/DockerfileSlim*`

* Add `get_annotation_python` and rename `get_annotation` to `get_annotation_ts`

* Add option to fallback to pip-compile

Put `# no_uv` on top the file for specific python script

Or set `USE_PIP_COMPILE` variable to `true`

* Put back `pip-tools` into shell.nix

* Make sure lockfile resolves again if `#no_uv` used

Add #no_uv to the end of requirements (requirements.in)
That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

* Put `pip install pip-tools` in original spot

* Fix compilation error

* Fix EE compilation error

error[E0658]: attributes on expressions are experimental
   --> windmill-worker/src/python_executor.rs:144:5
    |
144 |     #[cfg(feature = "enterprise")]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information

* Add `no_cache` annotation

Will force recalculation of lockfile
And block uv from using cached values

* Target uv cache to /tmp/windmill/cache

* Prohibit uv from managing python

* Add uv to DockerfileBackendTests

* Pin uv version to 0.4.18 in Dockerfiles

* Dont put `#no_uv` in requirements.in

Instead postfix hash for requirements.in with `-no_uv`

* Push Warning to logs if fallbacked to pip-compile

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 16:00:35 +02:00
Alexander Petric
6eb7f8ce81 feat(worker): support workers to run natively on windows (#4446)
* minimal code change to get windmill worker on windows for bun and python + rustfmt

* adding support for powershell

* compiling error on unix

* rust linting comments

* comments hugo: PSModulePath

* comments ruben, refactor to simplify

* adding build workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* skip migration env, ee fixes

* improvements powershell

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* killing process tree in windows

* sqlx_offline

* install openssl for github windows runner

* used pre-installed openssl

* used pre-installed openssl

* build ee

* build ee

* build ee

* build ee

* adding commented out steps for artifact publishing

* build on tag matchinv v* pattern

* ren instead of mv on Windows

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix merging issue

* gate imports for windows

* fixing default cargo home path...

* fixing default cargo home path...

* comments ruben

* make pwsh default modules loading more robust on unix (#4448)

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 15:59:12 +02:00
Ruben Fiszel
bd6ddc166b feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
HugoCasa
ae8f29a4cb feat: http routing (#4339)
* feat: http routing

* all

* feat: improve UI

* final stuff

* fix: sqlx

* fix: nit

* fix: nits

* fix: error handler display

* fix: routes panel perms

* all

* fix: improve ability to paste from macos in vscode extension

* fix lock-write for deno

* all

* cleaning

* fix

* cli preprocessor

* nits

* nits

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-20 15:38:38 +02:00
Ruben Fiszel
e1223a6f24 fix: cache hub scripts in more cases + pre-cache hub scripts deps in deno 2024-08-27 02:30:54 +02:00
Ruben Fiszel
2b3e86e341 feat: add vim support for monaco/webeditor 2024-08-26 22:31:23 +02:00
Ruben Fiszel
211e80b69e fix: make result_json path stable for python executors 2024-07-23 17:39:40 +02:00
Ruben Fiszel
be25101377 fix: add WM_SCHEDULED_FOR to contextual variables and early stop of flows 2024-07-18 02:10:15 +02:00
Ruben Fiszel
2812792326 fix: memory optimization for flow with big args (#4019) 2024-07-04 10:08:42 +02:00
Ruben Fiszel
0ad858ee5c fix: improve support of relative paths in subflows and apps 2024-06-21 18:07:58 +02:00
Ruben Fiszel
c8915063c4 feat: track dependency map for bun + inline script of flows 2024-06-08 16:43:48 +02:00
Ruben Fiszel
157e4d8e53 improve bun dedicated worker to use nodejs 2024-05-01 02:10:58 +02:00
Ruben Fiszel
b5db06d742 add base_internal_url to dedicated python workers 2024-04-30 22:48:40 +02:00
Ruben Fiszel
1cac7bf239 more logs for dedicated workers 2024-04-30 17:19:34 +02:00
Ruben Fiszel
2cb1595cae fix(python): supports folders starting with numbers for execution 2024-04-30 11:23:48 +02:00
wendrul
c3325c2cbf Instrument logs (#3566)
* Add spans to show worker_name and job_id

Always show worker_name and job_id when relevant, by creating spans that
live as long as the worker and the job.

* Remove redundant worker names from fn run_worker

Remove the now redundant worker_name tracing, as the information now
lives at the span level

* Use span when spawning new thread

* Instrument more pertinent functions

Move spans to more pertinent functions. Remove more redundant info, and
change worker_name to just worker

* Change tracing subscriber to log path:line_no

Change from logging target (module name usually) to file path and line
number
2024-04-18 01:11:14 +02:00
Ruben Fiszel
d63f5ee26e fix: add support for azure storage on distributed cache/logs 2024-03-23 18:42:38 +01:00
HugoCasa
6b719fecea feat: add workspace environment vars (custom contextual vars) (#3455)
* feat: custom contextual variables

* chore: sqlx prepare

* fix: main merge
2024-03-23 11:40:55 +01:00
Ruben Fiszel
7714221f3e feat: large log disk and distributed storage compaction 2024-03-23 10:55:29 +01:00
Ruben Fiszel
f8d3ba12af generalize cache object store 2024-03-21 08:50:10 +01:00
Ruben Fiszel
d616735174 fix compile on non ee 2024-03-20 15:34:25 +01:00
Ruben Fiszel
7dafa3d62a fix compile on non ee 2024-03-20 15:09:28 +01:00
Ruben Fiszel
a0a42067fb feat: s3 cache config added to instance settings + parallelized tar pulling 2024-03-20 14:19:34 +01:00
Ruben Fiszel
ab18058405 fix: allow multiple PIP_SECRETS 2024-03-17 15:15:36 +01:00
Ruben Fiszel
958df4f62c feat: refactor logging to improve performance by order of magnitude for longer jobs 2024-03-16 17:55:34 +01:00