* Implement MVP of Parallel uv installation
* Implement PY_CONCURRENT_DOWNLOADS
* Remove Flock for uv installs
* Make S3 pull/push parallel
* Refactor and allow to Cancel installation
* Dont print S3 in output if disabled
* Implement better error handling
* Polishing
* More polishing
* Implement error-handler for kill_tx_2.send()
* Fix and Format prev merge
* Presubscribe to all kill_tx's
We do it now before first event could fire
Meaning no events can be lost anymore
* Early print errors and safer error handling
* Return Err if installation failed
* Final changes
* Return error instead of just printing it
* Safer the way to acquire permit
* Fix compilation error
* Remove double error logs
* (python) Initially implement merge
Iterate overall all python paths and if same folder has same name multiple times,
then merge the content and put to <job_dir>/site-packages
Solves problem with imports for some dependencies.
Default layout (/windmill/cache/):
dep==x.y.z
└── X
└── A
dep-ext==x.y.z
└── X
└── B
In this case python would be confused with finding B module.
This function will convert it to (/<job_id>):
site-packages
└── X
├── A
└── B
This way python has no problems with finding correct module
* Better error-handling
* Optimize and fix bug
* Update backend/windmill-worker/src/python_executor.rs
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* Update backend/windmill-worker/src/python_executor.rs
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* Polish
* Add path to PathBuf
* Add & before PathBuf
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* feat: Handle `pip install` by `uv`
Dirty and untested, but already something working
* Integrate with NSJAIL and prepare fallbacks
* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific
* Remove `--disable-pip-version-check`
Reason:
warning: pip's `--disable-pip-version-check` has no effect
* Fix backend compilation error
* Pip fallback overwrite UV's cache
* Initially refactor cache (No S3)
* Support S3
* Remove unused import
* Handle flags for NSJAIL
* Return deleted flag
* Update Dockerfile
* Update docker-image.yml
* Update docker-image.yml
* Add --link-mode=copy and remove -v
* Fix NSJAIL INDEX_URL
* Fix flock and windows
* Update python_executor.rs
* Remove line from Dockerfile
We dont need it and to trigger build
* fixing for windows
* Dont pin python to specific version
* Change TMP for windows
* Revert docker-image.yml
* Disable UV for ansible
Will be enabled later.
Needs proper testing and its better to split onto 2 PRs with first modifying python and second ansible
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Alexander Petric <petric.al@gmail.com>
* Refactor annotation system
* Move tests to different location
* Limit annotations per line (15)
* Write initial benchmark
* Optimize
* More optimizations
* Rewrite with 3x speed!
* Remove benchmarks
They were in wrong place and for development
* Polishing
* Remove unused leftover
* Integrate new annotation system
* Add comment explanation
* Remove `hashe-based matching`
Turned to be it is impure and non-deterministic method
* Add `tracing::error` for `// native` without `deno_core`
If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.
This commit prints error to logs
* add more tests
* Update annotations.rs
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* Update `shell.nix`
- Replace pip-compile with uv packages
- Pin rust version
- Add var to trigger windmill print more info in stdout
* Replace `pip-compile` with `uv` (dirty + untested)
* Fix arguments passed to uv
Some of the flags are included by default in UV and can be safely removed:
- --resolver=backtracking
- --no-emit-index-url
Also uv does not support `--pip-args` and suggests to directly pass args to uv.
* Remove extra `dbg!`
* Replace 'pip-compile' with 'uv' in Dockerfile
* Add fallback option to `pip-compile` (Disabled)
* Add `uv` to `docker/DockerfileSlim*`
* Add `get_annotation_python` and rename `get_annotation` to `get_annotation_ts`
* Add option to fallback to pip-compile
Put `# no_uv` on top the file for specific python script
Or set `USE_PIP_COMPILE` variable to `true`
* Put back `pip-tools` into shell.nix
* Make sure lockfile resolves again if `#no_uv` used
Add #no_uv to the end of requirements (requirements.in)
That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved
* Put `pip install pip-tools` in original spot
* Fix compilation error
* Fix EE compilation error
error[E0658]: attributes on expressions are experimental
--> windmill-worker/src/python_executor.rs:144:5
|
144 | #[cfg(feature = "enterprise")]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information
* Add `no_cache` annotation
Will force recalculation of lockfile
And block uv from using cached values
* Target uv cache to /tmp/windmill/cache
* Prohibit uv from managing python
* Add uv to DockerfileBackendTests
* Pin uv version to 0.4.18 in Dockerfiles
* Dont put `#no_uv` in requirements.in
Instead postfix hash for requirements.in with `-no_uv`
* Push Warning to logs if fallbacked to pip-compile
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* minimal code change to get windmill worker on windows for bun and python + rustfmt
* adding support for powershell
* compiling error on unix
* rust linting comments
* comments hugo: PSModulePath
* comments ruben, refactor to simplify
* adding build workflow
* editing workflow
* editing workflow
* editing workflow
* editing workflow
* editing workflow
* skip migration env, ee fixes
* improvements powershell
* testing windows runner
* testing windows runner
* testing windows runner
* testing windows runner
* testing windows runner
* install postgres on runner
* install postgres on runner
* install postgres on runner
* install postgres on runner
* install postgres on runner
* install postgres on runner
* install postgres on runner
* install postgres on runner
* killing process tree in windows
* sqlx_offline
* install openssl for github windows runner
* used pre-installed openssl
* used pre-installed openssl
* build ee
* build ee
* build ee
* build ee
* adding commented out steps for artifact publishing
* build on tag matchinv v* pattern
* ren instead of mv on Windows
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix merging issue
* gate imports for windows
* fixing default cargo home path...
* fixing default cargo home path...
* comments ruben
* make pwsh default modules loading more robust on unix (#4448)
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* Add spans to show worker_name and job_id
Always show worker_name and job_id when relevant, by creating spans that
live as long as the worker and the job.
* Remove redundant worker names from fn run_worker
Remove the now redundant worker_name tracing, as the information now
lives at the span level
* Use span when spawning new thread
* Instrument more pertinent functions
Move spans to more pertinent functions. Remove more redundant info, and
change worker_name to just worker
* Change tracing subscriber to log path:line_no
Change from logging target (module name usually) to file path and line
number
