* fix(python): fix uv can't find ssl certificates
- Add `PY_NATIVE_CERT` flag, forces UV to use native tls
- Rename `PIP_INDEX_CERT` to `PY_INDEX_CERT`
- Rename `PIP_TRUSTED_HOST` to `PY_TRUSTED_HOST`
For backwards compatibility PIP* variables are still accessible
* feat(python): add `custom_wheels` directory to PYTHONPATH
Add global directory by path `<CACHE_DIR>/python_xyz/custom_wheels`
For example for scripts running python 3.11, in every execution
`<CACHE_DIR>/python_311/custom_wheels` will be accessible and all wheels placed there could be imported and used.
This is usefull for preinstalling wheels before runtime
* Make it work with Nsjail
* Rework and make custom_wheels optional
* Remove `create_dir_all` from imports
* Use sync version of metadata
* Rename `custom_wheels` to `global-site-packages`
- Add `PY_NATIVE_CERT` flag, forces UV to use native tls
- Rename `PIP_INDEX_CERT` to `PY_INDEX_CERT`
- Rename `PIP_TRUSTED_HOST` to `PY_TRUSTED_HOST`
For backwards compatibility PIP* variables are still accessible
* feat: Handle `pip install` by `uv`
Dirty and untested, but already something working
* Integrate with NSJAIL and prepare fallbacks
* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific
* Remove `--disable-pip-version-check`
Reason:
warning: pip's `--disable-pip-version-check` has no effect
* Fix backend compilation error
* Pip fallback overwrite UV's cache
* Initially refactor cache (No S3)
* Support S3
* Remove unused import
* Handle flags for NSJAIL
* Return deleted flag
* Remove verbose mode and enable link-mode=copy
* Granural migration of lockfiles
Before i realized we dont need it :)
* Initial draft (not-working)
* Add fallback
* Fix bug preventing uv from installing deps
'\n' - Love it
* Add verbosity indicator
* Iterate on feature
- Added instance python version
- Rework logic
* Fix EE build error
error[E0599]: no method named `iter` found for tuple `(PyVersion, std::vec::Vec<std::string::String>)` in the current scope
* Support S3
* Support NSJAIL
* Refactor `get_python`
* Make NSJAIL work [Unsafe]
config file missed /proc mount causing install phase to fail
* Trigger CI
* Clean up
* Make Actions build it
* Trigger CI #2
* Update Dockerfile and clean up
* Change fallbacks
now there is only no_uv and NOUV
* Expose INSTANCE_PYTHON_VERSION through env variable
* Change namings
* Include py-version to requirements.in
Also add comments and make code much cleaner
* Use const for python installation dir
It was hardcoded before
* Pin preinstalled version
* Update python_executor.rs
* Up to date branch
* Create PYCACHE dirs
TODO: PY_TAR_DIRS
* Fix after merge
* Make it safer
* Implement USE_SYSTEM_PYTHON
* Implement latest_stable option
* Load INSTANCE_PYTHON_VERSION on startup
* Check for multiple annotations used
* Fix Latest Stable button not pressed if selected
* Proper error handling for conflict on multiple annotations
* Fix merge conflicts
* Preinstall 3.11 and Latest Stable
* Preinstall latest stable in non-blocking manner
* Fix Warning
* Gate preinstall logic behind "python" feature
* Handle raw_deps properly
* Make it work with nsjail
* Revert docker-image.yml
* Revert Dockerfile
* Cleanup + Fixing
* Add windows support
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* Revert solution with Mutex
* Implement valid.windmill logic
* Remove unused import
* valid.windmill -> .valid.windmill
Just like .lock
* Dont delete wheels if cancelled/failed
Now we dont clean up requirement folder if it was failed.
This way we can fully utilize uv's flock system.
Also if we left wheel dir, but it was partially filled (resulting to invalid wheel)
we use --reinstall flag in order to overwrite any content of wheels
* Add comment
* Add --reinstall to nsjail
* fix(python): Cancel installation and start again within 1s caused module not found
* fix(python): Fix uv install job not exiting on fail
Specifically with nmslib installation was hanging without any output, even tho library build was failed.
It can be monitored with strace or catp.
* Replace ofiles approach with mutex
* Small refactor
* Return space for consistency
* Revert incorrect fix
* Fix module not found
* Remove unused import
* feat(cache): remove persistent raw values from queue
* test: add tests and fix discovered regressions
* fix csharp executor after rebase
* invalidate script cache after locking
* fix lite flow version race
* fix flow dependencies lock race
* update version flag to `v.1.440`
* fix no_uv not affecting deploy
Before this fix no_uv, no_uv_compile and no_uv_install were not affecting Dependency jobs
These jobs are only affected if used USE_PIP_COMPILE or USE_PIP_INSTALL env variables
To make it more consistant, no_uv should also affect dep jobs.
Also make ansible use uv by default
* Make it build
* Add no_uv_usage stats
* Provide build-env for Samael through shell.nix
* Run update_sqlx.sh
* Update ee-repo-ref
* Display memory peak while installing wheels with uv
* Invert DISABLE_NSJAIL
* Make it safer
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* fix no_uv not affecting deploy
Before this fix no_uv, no_uv_compile and no_uv_install were not affecting Dependency jobs
These jobs are only affected if used USE_PIP_COMPILE or USE_PIP_INSTALL env variables
To make it more consistant, no_uv should also affect dep jobs.
Also make ansible use uv by default
* Make it build
* Add no_uv_usage stats
* Provide build-env for Samael through shell.nix
* Run update_sqlx.sh
* Update ee-repo-ref
* Update sqlx cache
* Update ee-repo-ref
* Update ee repo ref
* Update sqlx cache
* Update ee-repo-ref.txt
* Implement MVP of Parallel uv installation
* Implement PY_CONCURRENT_DOWNLOADS
* Remove Flock for uv installs
* Make S3 pull/push parallel
* Refactor and allow to Cancel installation
* Dont print S3 in output if disabled
* Implement better error handling
* Polishing
* More polishing
* Implement error-handler for kill_tx_2.send()
* Fix and Format prev merge
* Presubscribe to all kill_tx's
We do it now before first event could fire
Meaning no events can be lost anymore
* Early print errors and safer error handling
* Return Err if installation failed
* Final changes
* Return error instead of just printing it
* Safer the way to acquire permit
* Fix compilation error
* Remove double error logs
* (python) Initially implement merge
Iterate overall all python paths and if same folder has same name multiple times,
then merge the content and put to <job_dir>/site-packages
Solves problem with imports for some dependencies.
Default layout (/windmill/cache/):
dep==x.y.z
└── X
└── A
dep-ext==x.y.z
└── X
└── B
In this case python would be confused with finding B module.
This function will convert it to (/<job_id>):
site-packages
└── X
├── A
└── B
This way python has no problems with finding correct module
* Better error-handling
* Optimize and fix bug
* Update backend/windmill-worker/src/python_executor.rs
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* Update backend/windmill-worker/src/python_executor.rs
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* Polish
* Add path to PathBuf
* Add & before PathBuf
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
