Commit Graph

455 Commits

Author SHA1 Message Date
Ruben Fiszel
628ae617f7 feat: docker runtime 2024-12-09 18:02:18 +01:00
Ruben Fiszel
c6dc19a893 improve otel 2024-12-08 11:06:35 +01:00
Ruben Fiszel
a40cc61619 add hostname to logs 2024-12-08 09:49:51 +01:00
Ruben Fiszel
324def26e3 add tls to tonic 2024-12-08 00:34:46 +01:00
Ruben Fiszel
8a5ba115c6 feat: add otlp support (#4869)
* all

* all

* add otel

* add otel

* update docker-image

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
2024-12-07 16:29:20 +01:00
Lucas Abel
ef08fc8e29 feat: add db storage for app inline scripts (#4837) 2024-12-06 13:37:45 +01:00
Lucas Abel
b66a31a623 feat(cache): refurbish fs backed cache (#4863)
* feat(cache): refurbish fs backed cache

* add `cached` exemple
2024-12-06 12:26:36 +01:00
Ruben Fiszel
b4011be2dc improve retries max value handling 2024-12-06 01:03:46 +01:00
Lucas Abel
a09d009e12 nit: format cache.rs (#4845) 2024-12-05 11:10:30 +01:00
Ruben Fiszel
19946deb82 simplify warn_after_seconds 2024-12-04 18:28:28 +01:00
Ruben Fiszel
715d09c30d simplify warn_after_seconds 2024-12-04 18:24:31 +01:00
Ruben Fiszel
f6c4951877 improve transparency for slow queries 2024-12-04 17:30:58 +01:00
wendrul
6987a36846 feat: indexer extra settings + parallel downloads + many improvements (#4822)
* Nits on the service logs page

* Show all the hosts returned by query + sumOtherDocCount warning

* Remove from index endpoint

* Fix tests

* Prepare sqlx
2024-11-29 19:14:05 +01:00
Lucas Abel
0a65b90049 feat(cache): implement flow node caching (#4808)
* feat(cache): implement flow node caching

* feat(cache): implement script caching

* feat(cache): improve cache
2024-11-29 09:35:00 +01:00
wendrul
db4172439c fix: service logs: aggregation query + killpill on index pull + retention period for logs on s3 (#4795)
* Search across all hosts in index. killpill on index pull

* Add retention period for service logs

* Remove console log

* monitor log_file when older than biggest retention period (job / log)

* Retention period deletes service and job logs

* Add setting to monitor logs out of s3

* Prepare sqlx

* Update ee repo ref

* Prepare sqlx

* Unused variables
2024-11-27 18:57:23 +01:00
Lucas Abel
4e52e54314 feat(backend): store flow inner modules into flow_node table (#4778) 2024-11-27 16:08:31 +01:00
Lucas Abel
60f2e9e242 enable is_simple optimization for FlowScript (#4799)
* backend: remove some allocations in `worker_flow.rs`

* feat(backend): enable `is_simple` optimization for `FlowScript`

* fix: fix node retrieval for `flowscript` inner step
2024-11-27 10:06:19 +01:00
pyranota
41c8a22830 fix(python): Merge to site-packages packages with same name (#4793)
* (python) Initially implement merge

Iterate overall all python paths and if same folder has same name multiple times,
then merge the content and put to <job_dir>/site-packages

Solves problem with imports for some dependencies.

Default layout (/windmill/cache/):

dep==x.y.z
└── X
   └── A
dep-ext==x.y.z
└── X
   └── B

In this case python would be confused with finding B module.

This function will convert it to (/<job_id>):

site-packages
└── X
   ├── A
   └── B

This way python has no problems with finding correct module

* Better error-handling

* Optimize and fix bug

* Update backend/windmill-worker/src/python_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/python_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Polish

* Add path to PathBuf

* Add & before PathBuf

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2024-11-26 18:24:26 +01:00
Lucas Abel
ce121fcf37 feat(backend): implement flow scripts (#4748)
* feat(backend): implement flow scripts

* conditionally resolve reduced flow value

* conditionally use flow scripts based on the min worker version

* grant all to `flow_{version_lite,node}`

* do not silently discard errors

* env: add flow script feature flag

* flow script on cloud automatically

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-11-25 23:20:51 +01:00
Lucas Abel
44997cea78 backend: keep tracking minimal worker version (#4786)
Be sure to also track the min version of workers that have been started
after the min version variable has been set. e.g. All workers are up to
date and `MIN_VERSION_IS_AT_LEAST_1_427` is set to true, a worker is
then powered on with an anterior version, leading to a false positive
value. This commit fix that.
2024-11-25 08:08:16 +01:00
wendrul
9f34712bd7 feat: Add a devops role to act as a "readonly admin" (#4775)
* Prepare sqlx

* Add devops role as bool

* Add devops reloe and use it for certain endpoints

* Critical alerts: complete devops role visiblity

* Update ee repo ref

* Prepare sqlx
2024-11-22 14:55:48 +01:00
Alexander Petric
54b419d8ab feat(schedule): support for extended cron syntax (#4754)
* init

* croner with fall back on scheduling failure

* adding packages

* fallback mechanism v1

* removing debug console log

* improving cargo organization

* queue cargo.toml

* use workspace cargo-tz

* test next occurrence when creating new croner schedule

* toggle label

* treat legacy edit requests where no cron_version is provided as v1

* fix edit schedule
2024-11-21 19:24:38 +01:00
Lucas Abel
002f0b92ea backend: rework raw values fetching logic (#4770) 2024-11-21 13:59:42 +01:00
wendrul
6e70df1db8 feat: Indexer improvements: s3 backup logic reworked, settings on the frontend (#4763)
* Add settings page + common code for indexer

* Prepare sqlx
2024-11-20 22:40:36 +01:00
Ruben Fiszel
f05064a4c8 add FORCE_HOSTNAME env var 2024-11-20 16:07:24 +01:00
HugoCasa
bb185e7318 refactor sql result formatting (#4760) 2024-11-20 14:53:39 +01:00
HugoCasa
1453b9bfe6 feat: pdf file preview (#4753)
also fixes s3 missing content-length header
2024-11-19 19:09:20 +01:00
Lucas Abel
b5f411a8ce feat(backend): move some static fields out of job tables (#4689)
* feat(backend): move some static fields out of job tables

* backend: re-use `flow_value` if any and avoid clone

* update minimal version to 1.427.0
2024-11-19 11:27:55 +01:00
wendrul
1e3b825b3e fix: playbook files permission mode incompatible with Windows (#4740) 2024-11-18 22:14:19 +01:00
wendrul
6f4a6d8a72 feat: Add mode (permissions) option to files in ansible (#4724)
* Add chmod option to files in ansible

* Add 0 prefix if string

* Rename chmod to mode
2024-11-18 17:02:38 +01:00
Alexander Petric
366149ff65 cloud_hosted non ee scope (#4723) 2024-11-15 18:14:11 +01:00
Alexander Petric
799ebf16c7 if CLOUD_HOSTED and workspace_id set, autoacknowledge for superadmin (#4719) 2024-11-15 16:31:42 +01:00
Alexander Petric
5b39f8f0ba feat(monitoring): workspace critical alerts (#4684)
* critical alert ui

* updating ui, backend logic

* revert

* type check fix npm

* checking out cli files from main

* moving alert icon

* adding sqlx mock data

* more sqlx changes

* feat(frontend): nodes from flow can be connected directly in expr input through a plug icon (#4652)

* Add flow prop picker

# Conflicts:
#	frontend/src/lib/components/propertyPicker/PropPicker.svelte

* fix unwanted copy

* cleaning

* Fix unset context

* move button and always display input

* fix unwanted proppicker display

* update

* update

* clean all

* clean all

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>

* replace hide/show with toggle

* adding mutable setting and navigation to settings to configure channels

* merge fix

* ee non ee changees

* auto-acknowledge when muted

* pr comments

* fix bad log

* user inner modal component

* update unaknowledge alerts after acknowledging from modal

* aknowledge -> acknowledge

* format

* adding backend support for workspace critical alerts

* immediately check for alerts

* immediately check for alerts

* adding openapi changes

* simplify loading of superadmin/ee

* update modal logic

* frontend logic update

* sqlx prepare

* adding mute functionality for workspace critical alerts + show alerts button for instance and workspace settings

* npm / rust warnings & errors

* reverting non-mac cargo toml

* reverting non-mac cargo toml

* mute toggles in critical alert ui

* ui polish

* adding acknowledged_workspace column

* sqlx prepare

* make sure we wait for all stores to be loaded

* adding workspace mute logic to report_critical alert

* auto ack recovered alerts

* toggle workspace as superadmin and critical alerts in logs menu

* adding critical alert button if width < 786px

* refresh on change

* Move notification to logs (#4698)

* Move notification to logs

* removing unused button

---------

Co-authored-by: Alexander Petric <petric.al@gmail.com>

* tailwind typo

* auto ack on mute

* don't change deref

* sqlx prep

* avoid renaming db column, keep acknowledged instead of acknowledged_global

* hide critical alert menu when not ee

* if workspace muted, also acknowledge global when workspace set

* removing save button for mute setting and improve ux/responsiveness

* fix: deployment callbacks have a concurrency limit of 1 on same path

* sqlx prep

* ee-repo ref

* z-[9999] for modal

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-11-15 09:48:48 +01:00
pyranota
04e2ef5809 feat: Handle pip install by uv (#4517)
* feat: Handle `pip install` by `uv`

Dirty and untested, but already something working

* Integrate with NSJAIL and prepare fallbacks

* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific

* Remove `--disable-pip-version-check`
Reason:
   warning: pip's `--disable-pip-version-check` has no effect

* Fix backend compilation error

* Pip fallback overwrite UV's cache

* Initially refactor cache (No S3)

* Support S3

* Remove unused import

* Handle flags for NSJAIL

* Return deleted flag

* Update Dockerfile

* Update docker-image.yml

* Update docker-image.yml

* Add --link-mode=copy and remove -v

* Fix NSJAIL INDEX_URL

* Fix flock and windows

* Update python_executor.rs

* Remove line from Dockerfile

We dont need it and to trigger build

* fixing for windows

* Dont pin python to specific version

* Change TMP for windows

* Revert docker-image.yml

* Disable UV for ansible

Will be enabled later.
Needs proper testing and its better to split onto 2 PRs with first modifying python and second ansible

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Alexander Petric <petric.al@gmail.com>
2024-11-15 00:12:56 +01:00
dieriba
bcc96d7e6a feat: Support mistral anthropic for ai (#4692)
* wip: openai proxy and other ai proxy integration

* fixing migration script

* wip: support different ai provider in front, fix proxy openai

* wip: adding frontend ai provider

* updated copilot types

* wip: working on anthropic integration

* done AI proxy front

* adding new type and support for anthropic

* updating gitignore

* adding streaming response

* added streaming prompt

* push lib/gen

* wip: fixing anthropic

* anthropic fully supported

* fix backend missing var error and fully support stream event for anthropic

* remove gen directory

* fixing openapi file

* add support for mistral, and update create workspace components

* remove deref.json

* remove package-json

* openapi

* fix ui enable code

* added utility function for init workspace ai provider

* fix workspace switch bug

* update anthropic property and fixed frontend error

* fix workspace settings

* update error message and fix typo migration file

* chore: update openapi file

* fix dev file

* add .sqlx

* all

* update sqlx

---------

Co-authored-by: dieriba <t.dieriba@gmail.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-11-14 14:24:35 +01:00
Lucas Abel
47424b1446 feat(backend): monitor minimal version of living workers (#4704) 2024-11-14 12:33:50 +01:00
Lucas Abel
231dcbb470 api: cleanup job data structure (#4705) 2024-11-14 12:25:30 +01:00
Lucas Abel
d7b688d352 nit: generalize usage of has_failure_module (#4706) 2024-11-14 08:49:12 +01:00
Alexander Petric
f870518d94 feat(frontend): critical alerts UI (#4653) 2024-11-09 00:42:10 +01:00
HugoCasa
c686cc437d feat: http custom routes for static assets (#4666)
* feat: http custom routes for static assets

* update ee ref + fix build

* fix build

* nit
2024-11-07 20:16:14 +01:00
HugoCasa
4da8726eee allow no tls for instance settings smtp (#4656)
* allow no tls for instance settings smtp

* update ee ref
2024-11-06 16:58:24 +01:00
Guilhem
cc1d3f2193 feat(frontend): improve and simplify scheduled poll flows (#4560) 2024-11-04 23:18:09 +01:00
Ruben Fiszel
fd4fecf945 split better oss and ee 2024-11-02 16:31:03 +01:00
HugoCasa
c803f9cdb2 feat: private hub user accessible url setting (#4617)
* feat: private hub user accessible url setting

* improve UI
2024-11-01 11:20:31 +01:00
Ruben Fiszel
557b949efe fix(s3): align s3 handler additional creds providers 2024-11-01 00:51:28 +01:00
HugoCasa
7fdb524fc6 improve key renewal (#4519)
* improve key renewal

* sqlx and fix

* fix build

* improve everything

* nits

* nits

* update ee ref

* nit
2024-10-29 09:26:45 +01:00
Ruben Fiszel
71f84dee08 handle better empty tag overrides 2024-10-29 09:01:02 +01:00
Ruben Fiszel
ab38215c86 fix(backend): in flows, workspace scripts should use their set tags instead of the default one 2024-10-29 07:29:39 +01:00
Ruben Fiszel
85de44cb93 feat: autoscaling v0 (#4593)
* all

* all

all

all

all

all

all

all

all

all

* all

* all

* nits

* nits
2024-10-28 00:47:19 +01:00
wendrul
e645e6fe5e Remove debug line (#4548) 2024-10-18 20:31:53 +02:00