Commit Graph

484 Commits

Author SHA1 Message Date
Ruben Fiszel
8e0647833a fix: hide values of WHITELIST_ENVS 2025-02-04 22:11:05 +01:00
Ruben Fiszel
8e5ab4d40a nit fix hub cache init on server 2025-02-01 08:55:44 +01:00
Ruben Fiszel
d26c7e0913 nit error handling cache 2025-02-01 07:48:43 +01:00
Ruben Fiszel
af87d635fb use sync decrypt 2025-01-31 10:31:19 +01:00
Ruben Fiszel
e1fb537b5d fix: fix codebase script execution 2025-01-30 19:06:55 +01:00
Ruben Fiszel
8004c5da71 fix: improve codebase support 2025-01-30 18:25:08 +01:00
Ruben Fiszel
6ea28edc41 fix compile 2025-01-27 22:00:48 +01:00
dieriba
900a8d675e feat: postgres triggers (#4860)
* feat: init database triggers

* feat:  wip: database_triggers

* feat:  add database triggers front view

* feat: 🚧 database_triggers

* feat: 🚧 add definition in yaml, updated backend code and added migration

* feat: 🚧 updated migration file, update openapi.yml, updated database_triggers page and backend function

* fix: struct rust

* feat: 🚧 update migrate, database trigger backend function fixed

* feat: 🚧 add resource picker front, update backend function

* feat: 🚧 edit inner database inner, update triggers

* feat: 🚧 database_triggers

* feat: 🚧 update openapi yaml, prettied websocker trigger

* feat: 🚧 database_triggers

* feat: 🚧 add resource module, update variable file, working on main loop for database_trigger

* feat: 🐛 working sqlx query

* feat: 🚧 fix query with sqlx, added main loop

* feat: 🚧 add new column database_trigg

* feat: 🛂 run jobs works

* feat: 🚧 handling slot name and replication

* feat: restructring triggers, decoding trigger message on work

* feat: 🚧 database_trigger

* feat: 🚧

* feat: 🚧 converter done, work on custom script

* feat: 🚧 multiple trigger

* feat: 🚧 adding new argument function

* feat: 🚧 database_trigger

* feat: 🚧 add generate template for front, update script picker

* feat: 🚧 template script fix bug, work on restructing backend logic

* feat: 🚧 update autogenerated script, add persistence state for template script

* feat: 🚧 update structure client

* feat: 🚧 rewrited crud function

* feat: 🚧 added publication handler

* feat: 🚧 new ui finished

* feat: 🚧 added slot function hanlder finish front ux

* feat:  ux improvement done, backend logic done

* feat: 🐛 fix where clause

* feat:

* feat:

* chore: update .sqlx and remove empty package

* fix: update publication and remove unneccessary print

* feat: finish converter to json, remove save button, remove unused crate

* feat: update migration for database trigger, fixed query, fixed frontend error with missing feature on back

* chore: update .sqlx

* chore: update sqlx

* chore: .sqlx

* chore: add unused

* fix: use right database resource in back and front

* fix build

* refactor:

* fix sqlx

* nits: retrieve all script from database trigger editor

* fix: template script object, fix route name

* merge main

* feat: rework ux

* feat/finish ux and update backend logic

* feat: database_trigger

* feat: database_trigger

* chore: update sqlx

* chore: fix ci

* nits: fix worker.rs

* feat: database_trigger

* Update frontend/src/lib/components/triggers/database/DatabaseTriggersPanel.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* feat:

* fix ci

* nits: database to postgres

* nits: fix .sqlx

* feat: add link to original code and open tab functionallity

* nits: update .sqlx

* nits: fix request type and yaml file

* nits: add abel suggestion

* fix: update .sqlx

* fix: update .sqlx

* nits: add comment to hex.rs

* nits: remove features, rename database to postgres name

* nits: replace database name by postgres

* nits: database name to postgres

* nits: update .sqlx

* nits: update .sqlx

* nit

* nit

* nits

* nit

* nits

* nits

* nits

* nits

* fix sqlx

* nits: update .vscode

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-24 15:12:41 +01:00
pyranota
3af3fc898b feat(python): Multiple runtime versions (#4579)
* feat: Handle `pip install` by `uv`

Dirty and untested, but already something working

* Integrate with NSJAIL and prepare fallbacks

* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific

* Remove `--disable-pip-version-check`
Reason:
   warning: pip's `--disable-pip-version-check` has no effect

* Fix backend compilation error

* Pip fallback overwrite UV's cache

* Initially refactor cache (No S3)

* Support S3

* Remove unused import

* Handle flags for NSJAIL

* Return deleted flag

* Remove verbose mode and enable link-mode=copy

* Granural migration of lockfiles

Before i realized we dont need it :)

* Initial draft (not-working)

* Add fallback

* Fix bug preventing uv from installing deps

'\n' - Love it

* Add verbosity indicator

* Iterate on feature
- Added instance python version
- Rework logic

* Fix EE build error
error[E0599]: no method named `iter` found for tuple `(PyVersion, std::vec::Vec<std::string::String>)` in the current scope

* Support S3

* Support NSJAIL

* Refactor `get_python`

* Make NSJAIL work [Unsafe]

config file missed /proc mount causing install phase to fail

* Trigger CI

* Clean up

* Make Actions build it

* Trigger CI #2

* Update Dockerfile and clean up

* Change fallbacks
now there is only no_uv and NOUV

* Expose INSTANCE_PYTHON_VERSION through env variable

* Change namings

* Include py-version to requirements.in

Also add comments and make code much cleaner

* Use const for python installation dir

It was hardcoded before

* Pin preinstalled version

* Update python_executor.rs

* Up to date branch

* Create PYCACHE dirs

TODO: PY_TAR_DIRS

* Fix after merge

* Make it safer

* Implement USE_SYSTEM_PYTHON

* Implement latest_stable option

* Load INSTANCE_PYTHON_VERSION on startup

* Check for multiple annotations used

* Fix Latest Stable button not pressed if selected

* Proper error handling for conflict on multiple annotations

* Fix merge conflicts

* Preinstall 3.11 and Latest Stable

* Preinstall latest stable in non-blocking manner

* Fix Warning

* Gate preinstall logic behind "python" feature

* Handle raw_deps properly

* Make it work with nsjail

* Revert docker-image.yml

* Revert Dockerfile

* Cleanup + Fixing

* Add windows support

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-24 11:42:48 +01:00
Alexander Petric
5ad6c9e2d3 feat: msft teams support for critical alerts (#5113)
* feat: msft teams support for critical alerts

* ee changes

* ee

* sqlx prep

* multiple teams channels

* commit file, not symlink

* improve reactivity

* docs link

* Update ee-repo-ref.txt
2025-01-23 19:57:13 +01:00
HugoCasa
945bac6d64 fix memory limits reading on fly.io/firecracker (#5114) 2025-01-22 12:53:38 +01:00
Ruben Fiszel
abd06c017e fix: improve workspace specific tags behavior 2025-01-21 18:15:34 +01:00
Ruben Fiszel
ab9057aa0e fix: improve workspace specific tags behavior 2025-01-21 16:24:50 +01:00
Ruben Fiszel
c3daefda30 fix: fix workspace specific custom tags 2025-01-17 19:48:38 +01:00
HugoCasa
6b64485309 fix: preprocessor script kind option (#5081)
* fix: preprocessor script kind option

* oups

* oups
2025-01-17 14:10:39 +01:00
HugoCasa
ad5a2687a2 fix: improve runnable permissions (#5063) 2025-01-15 18:25:12 +01:00
HugoCasa
104e46990a fix(backend): better deleted user error (#5060)
* fix(backend): better deleted user error

* nit
2025-01-15 17:02:10 +01:00
wendrul
a8f49d5661 feat: add oracle db support on ee (#5062) 2025-01-14 21:21:24 +01:00
HugoCasa
175379fbe9 feat: on behalf of (#5058)
* feat: on behalf of

* fix

* revert mistakke

* temp ee ref

* renaming + CLI

* sqlx

* nits + ee ref
2025-01-14 18:32:25 +01:00
Lucas Abel
f7506da34c cache: clear the cache between tests (#4958)
otherwise tests aren't correctly isolated
2024-12-20 20:03:56 +09:00
Lucas Abel
12d18beb32 cache: better cache system (#4951)
* cache: consider source as untrusted

* cache: better scoped cache

* cache: non-lazy flow value parse error

* fixup! cache: consider source as untrusted

* fixup! cache: consider source as untrusted

* fixup! cache: better scoped cache
2024-12-20 11:19:57 +01:00
Ruben Fiszel
a857314f59 fix: fix relative imports in cached flow scripts 2024-12-18 11:34:07 +01:00
Lucas Abel
c798a6151f cache: disable caching for tests (#4943)
As `#[sqlx::test]` spawn a database per test, the cache
could yield unexpected results
2024-12-18 09:57:31 +01:00
Lucas Abel
73e6520312 cache: improve error reporting (#4940) 2024-12-17 19:44:51 +00:00
Lucas Abel
dab7b39917 feat(cache): remove persistent raw values from queue (#4866)
* feat(cache): remove persistent raw values from queue

* test: add tests and fix discovered regressions

* fix csharp executor after rebase

* invalidate script cache after locking

* fix lite flow version race

* fix flow dependencies lock race

* update version flag to `v.1.440`
2024-12-16 17:10:16 +01:00
Ruben Fiszel
e7d9f8ea63 split workspace and auth logic 2024-12-15 13:13:09 +01:00
Ruben Fiszel
fd462a7038 feature gates more feature for faster compilation (#4928)
* cleanup

* cleanup

* all

* all

* all

* all
2024-12-15 00:00:37 +01:00
Lucas Abel
c430c95886 cleanup is_flow usage and impl (#4923) 2024-12-13 18:08:01 +01:00
wendrul
6b27a0b91d feat: add C# support (#4908)
* Add csharp stuff and merge after the rewrite

    * Add csharp, boilerplate + minimal execution
    * Add initial parser
    * Frontend + wasm export of the tree sitter parser
    * Arg spread and use cache

* More types and req parsing

* Add logo

* Parse async and void to fit the wrapper to the main sig

* Csharp editor resource and var buttons

* Lockfile generation

* Add default parsing using serde_json::from_str

* Update init code for c#

* Nsjail for c#

* Remove c_variadic feature

* Remove unused imports

* Add c_variadic only for wasm32 assembly

* Fix c# test

* Add dotnet to the images

* Update dockerfiles

* Update dockerfile again

* Touch Dockerfile

* npm install c# parser

* Add opt mount in nsjail

* Update ee repo ref

* Add csharp to backend image for tests

* Add setting to set nuget package

* Windows compatibility

* Disable csharp test

* ADd feature flag
2024-12-13 00:29:01 +01:00
Lucas Abel
3f4e111c57 feat(cache): re-work job results cache (#4898) 2024-12-11 12:01:48 +01:00
Ruben Fiszel
f40fcf7b7e improve debug logs 2024-12-10 12:35:56 +01:00
Ruben Fiszel
628ae617f7 feat: docker runtime 2024-12-09 18:02:18 +01:00
Ruben Fiszel
c6dc19a893 improve otel 2024-12-08 11:06:35 +01:00
Ruben Fiszel
a40cc61619 add hostname to logs 2024-12-08 09:49:51 +01:00
Ruben Fiszel
8a5ba115c6 feat: add otlp support (#4869)
* all

* all

* add otel

* add otel

* update docker-image

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
2024-12-07 16:29:20 +01:00
Lucas Abel
ef08fc8e29 feat: add db storage for app inline scripts (#4837) 2024-12-06 13:37:45 +01:00
Lucas Abel
b66a31a623 feat(cache): refurbish fs backed cache (#4863)
* feat(cache): refurbish fs backed cache

* add `cached` exemple
2024-12-06 12:26:36 +01:00
Ruben Fiszel
b4011be2dc improve retries max value handling 2024-12-06 01:03:46 +01:00
Lucas Abel
a09d009e12 nit: format cache.rs (#4845) 2024-12-05 11:10:30 +01:00
Ruben Fiszel
19946deb82 simplify warn_after_seconds 2024-12-04 18:28:28 +01:00
Ruben Fiszel
715d09c30d simplify warn_after_seconds 2024-12-04 18:24:31 +01:00
Ruben Fiszel
f6c4951877 improve transparency for slow queries 2024-12-04 17:30:58 +01:00
wendrul
6987a36846 feat: indexer extra settings + parallel downloads + many improvements (#4822)
* Nits on the service logs page

* Show all the hosts returned by query + sumOtherDocCount warning

* Remove from index endpoint

* Fix tests

* Prepare sqlx
2024-11-29 19:14:05 +01:00
Lucas Abel
0a65b90049 feat(cache): implement flow node caching (#4808)
* feat(cache): implement flow node caching

* feat(cache): implement script caching

* feat(cache): improve cache
2024-11-29 09:35:00 +01:00
wendrul
db4172439c fix: service logs: aggregation query + killpill on index pull + retention period for logs on s3 (#4795)
* Search across all hosts in index. killpill on index pull

* Add retention period for service logs

* Remove console log

* monitor log_file when older than biggest retention period (job / log)

* Retention period deletes service and job logs

* Add setting to monitor logs out of s3

* Prepare sqlx

* Update ee repo ref

* Prepare sqlx

* Unused variables
2024-11-27 18:57:23 +01:00
Lucas Abel
4e52e54314 feat(backend): store flow inner modules into flow_node table (#4778) 2024-11-27 16:08:31 +01:00
Lucas Abel
60f2e9e242 enable is_simple optimization for FlowScript (#4799)
* backend: remove some allocations in `worker_flow.rs`

* feat(backend): enable `is_simple` optimization for `FlowScript`

* fix: fix node retrieval for `flowscript` inner step
2024-11-27 10:06:19 +01:00
pyranota
41c8a22830 fix(python): Merge to site-packages packages with same name (#4793)
* (python) Initially implement merge

Iterate overall all python paths and if same folder has same name multiple times,
then merge the content and put to <job_dir>/site-packages

Solves problem with imports for some dependencies.

Default layout (/windmill/cache/):

dep==x.y.z
└── X
   └── A
dep-ext==x.y.z
└── X
   └── B

In this case python would be confused with finding B module.

This function will convert it to (/<job_id>):

site-packages
└── X
   ├── A
   └── B

This way python has no problems with finding correct module

* Better error-handling

* Optimize and fix bug

* Update backend/windmill-worker/src/python_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/python_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Polish

* Add path to PathBuf

* Add & before PathBuf

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2024-11-26 18:24:26 +01:00
Lucas Abel
ce121fcf37 feat(backend): implement flow scripts (#4748)
* feat(backend): implement flow scripts

* conditionally resolve reduced flow value

* conditionally use flow scripts based on the min worker version

* grant all to `flow_{version_lite,node}`

* do not silently discard errors

* env: add flow script feature flag

* flow script on cloud automatically

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-11-25 23:20:51 +01:00
Lucas Abel
44997cea78 backend: keep tracking minimal worker version (#4786)
Be sure to also track the min version of workers that have been started
after the min version variable has been set. e.g. All workers are up to
date and `MIN_VERSION_IS_AT_LEAST_1_427` is set to true, a worker is
then powered on with an anterior version, leading to a false positive
value. This commit fix that.
2024-11-25 08:08:16 +01:00